DOCUMENT RESUME 



ED 294 497 



HE 021 439 



AUTHOR 
TITLE 



INSTITUTION 
PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Suskie, Linda A. 

Survey Research: What Works for the Institutional 
Researcher. Institutional Research Information Series 
No. 1. 

North East Association for Institutional Research. 
88 

102p. 

North East Association for Institutional Research, 
C/O Larry Metzger, Ithaca College, Ithaca, NY 14850 
($12.00). 

Guides - Non-Classroom Use (055) 
MF01/PC05 Plus Postage. 

Higher Education; Mnsti tutional Research; 
^Measurement Techniques; ^Questionnaires; ^Research 
Design; Sampling; "Statistical Analysis;; * Surveys 



ABSTRACT 

A guide to survey research is presented for both 
novice and experienced researchers. Steps of the survey research 
process are covered: (1) planning the survey to determine the purpose 
cf the study, collecting background information, designing the 
sample, and making a time line for completing the project; (2) 
questionnaire design, including the pros and cons of various question 
formats; (3) questionnaire development, including concepts such as 
maximizing response rate and establishing validity and reliability; 
(4) details of conducting a survey such as preparing a cover letter, 
pilot testing, and making followup mailings; (5) preparing the 
returned survey for data processing (editing and coding); (6) 
choosing the most appropriate statistical analysis for the data; and 
(7) reporting survey results. A question and answer format and an 
informal style have been used in the guide. Appendices provide: a 
sample time line for a survey research project, examples of 
questionnaires, and an example of a cover letter. (SW) 
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PREFACE 



This monograph is designed for institutional researchers 
and others interested in research in higher education. It 
has two purposes: first, to provide the novice with a guide 
to the basic steps of survey research, and, second, to pro- 
vide the more experienced researcher with a useful reference 
tool . 

This monograph grew from my interest in survey research 
in general and questionnaire design in particular. Many of 
the principles learned in my graduate studies in educational 
testing and measurement can be applied to questionnaire 
design. After finding myself repeatedly answering the same 
questions from colleagues on survey research, I wrote a short 
manual. The manual grew into a series of workshops I have 
presented around the country for the Association for Institu- 
tional Research and the North East Association for Institu- 
tional Research. Those workshops, in turn, inspired this 
monograph . 

This monograph has been written in an informal style to 
keep it readable and interesting. A question and answer 
format has been used throughout to help you focus quickly on 
the point of the discussion and use this monograph as a 
reference. Rather than use citations, I admit freely to 
borrowing the thoughts and findings of many; they are listed 
in the bibliography at the end. 

The monograph has been organized in a roughly chron- 
ological fashion to take you step-by-step through the survey 
research process. Chapter I discusses planning the survey: 
determining the purpose of the study, collecting background 
information, designing the sample, and making a time line for 
completing the project. Chapter II introduces questionnaire 
design by discussing the pros and cons of various question 
formats. Chapter III is the central chapter of the monograph. 
It examines developing the questionnaire itself, and it is 
here that such important concepts as maximizing response rate 
and establishing validity and reliability are introduced. 

Chapters IV through V deal more with the mechanics of 
conducting a survey. Chapter IV addresses such essential 
details as preparing a cover letter, pilot testing, and 
making follow-up mailings. Chapter V deals with preparing 
the returned surveys for data processing: editing and 
coding. 

Chapters VI and VII discuss analyzing and reporting your 
findings. Chapter VI is an attempt to cover several semesters 1 
course work in statistics in a few pages that serve as a 
guide to choosing the most appropriate statistical 
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analysis for your data and your needs. Those readers with 
some background in inferential statistics who need an easy 
reference guide should find it of value. Chapter VII 
discusses reporting the survey results. 

Aj though it mig^t be tempting to read this monograph "as 
you go 1 , it is strongly recommended that you go through the 
entire monograph and not just Chapter I as you plan your 
survey. Subsequent chapters will raise additional questions 
and ideas that will likely influence your plans. 

Deep gratitude is expressed to Sidney Micek, who con- 
ceived of this monograph and proposed the outline and format, 
editor Jennifer Presley for her assistance, Gary Reighard for 
his helpful comments on earlier drafts, my husband Steve for 
his suppoit and suggestions throughout this project, and the 
many institutional researchers around the country who have 
attended my workshops on questionnaire design, discussed 
their ideas and concerns, and shared examples of question- 
naires with me. 



3 



Chapter I. 
PLANNING THE SURVEY 



Any good survey is a major research project, involving 
considerable time and resources. To make sure your efforts 
pay off, it's crucial that you spend some time planning. A 
survey t^at doesn't provide needed information or delivers it 
too late to have an impact is a tremendous waste. This chap- 
ter will help you delineate the survey's objectives, make 
some basic data collection and analysis decisions including 
selecting a sample, and set deadlines so everything gets done 
in time . 



WHAT ARE THE OBJECTIVES OF YOUR SURVEY? 

State the objectives of your survey as specifically as 
possible. It's not enough to say you want to investigate 
student opinions on alcohol consumption. Refine the problem. 

Put your objectives in writing. If they're written 
down, they'll be clearer, easier to explain to others, and 
easier to keep in mind as you plan the survey. 



H0V7 WILL THE RESULTS BE USED? 

What decisions are to be made from the survey rerults? 
Surveys are not ends unto themselves. They are tools that 
are used to help make decisions. You wouldn't buy tools for 
a workbench unless you knew what you would use them for and 
unless you knew for sure that you w ere going to use them. 
Similarly, you shouldn't conduct a survey unless you know 
what decisions the survey data will be needed for and unless 
you are convinced that you really need survey data to help 
make those decisions. Talk to the people who requested the 
survey and to anyone else who might use the results. 

Sometimes, of course, a survey is needed for general 
background information and not to contribute to specific 
decisions. A periodic profile of the student body's goals 
and plans, for example, is helpful to the college adminis- 
tration and faculty in a wide range of contexts and not just 
for one or two specific decisions. In this case, you should 
still understand ^hy there is a perceived need for the survey 
information and what uses will be made of the survey results. 
Will it be worth the time and resources spent collecting the 
information? 
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WHAT ARE THE CRITICAL QUESTIONS TO BE ANSWERED? 

You may find that to meet all the objectives on your 
list, you'll have to ask dozens or hundreds of questions. 
You'll have drop some objectives from your list just to 
keep things from getting -coo complicated. Talk again to the 
people who will be using the survey results. Find out what 
the most crucial decisions or objectives are. What are the 
essential points that people want to find out no matter what? 
Why are they critical. -> 



WHAT CONCEPTS NEED TO BE DEFINED? 

To make sure your survey provides the information 
needed, some of the concepts used will doubtless need clar- 
ification. Terms such as "attrition," "non-traditional 
student," "degree-seeking," "teaching effectiveness," and 
even "satisfaction" and "quality" are full of ambiguities. 
If these types of concepts are applicable to your study, 
define them to everyone's satisfaction now , before the study 
gets underway. 



WHAT HAVE OTHERS DONE ON THIS TOPIC? 

Depending on the nature of the problem you are studying, 
a review of relevant literature and published survey instru- 
ments could be a considerable time-saver. Why reinvent the 
wheel when you can take advantage of what others have done 
before you? If the problem you're studying is student attri- 
tion, for example, a review of what others have found should 
help you hone in quickly on factors you should be examining. 
A review of surveys others have done on your subject will 
give you ideas on questions to ask and how to ask them. Try 
asking fellow institutional researchers at other colleges if 
you can adapt their questionnaires for your purposes rather 
than write your own from scratch. 



DO YOU REALLY NEEF A SURVEY? 

Ask yourself critically if you really need a survey to 
get the information you need. Perhaps the review of what 
others have done will suffice. Will your survey likely yield 
the same results that others have found? If so, is it worth 
the time and expense to conduct your own survey? Would a rew 
phone calls or interviews give you all you need? If all you 
need is factual data, are you sure it isn'': available from 
another source such as student records, tests, or direct 
observation? At the very least, can demographic information 
be obtained elsewhere? 
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WHAT INFORMATION DO YOU NEED TO ANSWER "HE QUESTIONS? 



The next step is to write a list of the specific infor- 
mation — facts, figures, and opinions — that you will need to 
find out to meet your objectives. Then ask yourself how each 
piece of information will contribute to your objectives. Be 
particularly critical of demographic information such as 
information on sex, age, racial or ethnic group, and marital 
status. Is it really important, for example, that you find 
out the sex of your respondents? Think about how you will 
use this information. Since men and women must be treated 
alike in many situations, your findings may be of no use and 
little interest to anyone. 



HOW WILL YOU ANALYZE AND REPORT THE DATA? 

What will you do with the information *fter you get it? 
Some constraints may be placed on your study depending on its 
purpose, computer availability, and the statistical sophisti- 
cation of your users. 

Is the purpose of your study to describe or explain? 
Many surveys are conducted simply to describe a group of 
people. Alumni surveys, for example, are often conducted to 
describe the alumni: their occupations, incomes, further 
education, residence, etc. Other surveys, however, are done 
to explain, predict, or explore . Why do some admitted fresh- 
men choose not to enroll at your school? What types of stu- 
dents are most likely to drop out? What factors contribute 
to alumni career paths? 

If your purpose is only to describe responses by summa- 
rizing them into totals and percents, the information you 
collect can be in almost any form. If you are not planning 
on a large number of responses, these simple tallies may be 
made by hand without the help of a computer. Larger surveys 
may be summarized on a personal computer. 

If you plan to explain, predict, or explore, however, 
you will probably need complicated statistical analyses such 
as factor analysis, regression analysis, or analysis of 
variance. These analyses require (1) knowledge of statistics 
either by ycu or by someone available to advise you, (2) a 
statistical software package such as SAS or SPSS and computer 
facilities large enough to handle it, and (3) data in a 
proper format. There are four basic data formats or "scales" 

Categorical or nominal data breaks people into cate- 
gories. Examples of categorical data include racial/ethnic 
group, marital status, major, and responses to many multiple 
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choice questions. Categorica- data cannot be ranked and 
means or medians cannot be calculated. They are difficult to 
use in statistical analyses designed to explain, predict, or 
explore. You should therefore avoid collecting this type of 
data if you want to do more than describe. 

Ranked or ordfe red data may be collected from questions 
asking for an "excellent/good/fair/poor" rating or similar 
ratings. Medians can be calculated but not means. This type 
of data can be analyzed using non-pararnetric statistical 
analyses or sufficiently robust parametric statistical ana- 
lyses. It therefore can be used to a limited extent to 
explain, predict, or explore. 

Scaled or interval data can be converted into meaningful 
numbers, where the difference between, say, a 1 and 2 is the 
same as the difference between a 4 and 5. Means can be cal- 
culated, and the data can be analyzed using a wide variety of 
powerful parametric statistical techniques. If you would 
like to use a fairly complex statistical analysis with your 
data, you should try to make it interval. 

Bivariate or dichotomous data are data that have only 
two possible values (e.g., sex, full-time/part-time status). 
Although they are technically categorical data, because there 
are only twc values they can be treated as interval data. 
Categorical and ordered data can often be grouped into bivar- 
iate categories and then introduced into a parametric statis- 
tical analysis. 

A final, question to consider is who will use your 
findings. Unsophisticated users may not be able to under- 
stand or appreciate a factor analysis, even if you determine 
that it would best meet their needs. Simple tallies may be 
all your users want. 



WHOM WILL YOU SURVEY? 

Your next step in planning your survey is to identify 
the group you want to survey. While this may seem a simplis- 
tic question, it requires careful thought. Suppose you are 
planning a survey of alumni. Do you want to include people 
who are in your school's alumni files but never completed a 
degree from your school? Do you want to include those who 
earned only a graduate degree from your school? Do you want 
to survey all alumni or just those of, say, the last five 
years? Suppose you are planning a survey of the student body 
on possible attrition causes. Do you want to survey all 
currently enrolled students or jus 4 * undergraduates? Or just 
full-time undergraduates? Or just full-time degree-seeking 
undergraduates? Or just full-time degree-seeking under- 
graduates who live o i campus? 
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This is also an important question because the type of 
people you survey will affect the kinds of information you 
can ask for. You cannot expect people with eighth grade 
educations, for example, to give you sophisticated opinions 
on complex economic issues. Teenagers will, by lack of 
experience , be uninformed on many consumer problems like 
credit and budgeting. 



DO YOU NEED A SAMPLE OR A CENSUS? 

Most surveys in higher education involving contacting a 
sample or subgroup rather than making a censu s or survey of 
everyone. The major reason for this is cost effectiveness. 
Contacting a smaller group will save postage, printing, and 
data processing costs as well as time in assembling mailings 
and editing and coding the results. If the sample is care- 
fully chosen and sufficiently large, these savings come with 
little sacrifice in terms of statistical accuracy. 

It f s also worth pointing out that a true census with a 
response from everyone is almost impossible to obtain. There 
will always be people with incorrect addresses and people who 
just plain refuse to cooperate. So attempting a census will 
likely bring only a sample of responses, albeit a fairly 
large one . 

There are two instances you might attempt to contact 
everyone rather than take a sample. The first is if your 
entire group is relatively small. If your entire student 
body is only 450 students or if you have only 150 computer 
science alumni, go ahead and survey them all. The second 
instance is if your survey is on a critical subject and 
someone influential might try to disparage the results 
because they are from "only" a sample. 



HOW MANY PEOPLE SHOULD YOU SURVEY? 

The number of people you survey depends on the amount of 
"sample error" you're willing to put up with. "Sample error" 
describes the possible difference between your findings and 
the true results if you were able to obtain valid responses 
from everyone . For example, suppose you found that 35% of 
your students were planning on graduate study with a "sample 
error" of 3%. This means you could be quite sure (actually 
95% sure) that between 32% and 38% of all your students are 
planning on graduate study (i.e., 35% - 3% and 35% + 3%). It 
makes intuitive sense that, the more people you survey, the 
smaller the sample error will be. The table on the next page 
lists the sample size needed for each amount of sample error. 



Random Sample Size 



Sample Error 



196 
264 
384 
600 



7% 
6% 
5% 
4% 
3% 
2% 
1% 



1,067 
2,401 
9,604 



While increased sample size reduces the sample error, 
you can see from the table that the benefits eventually 
diminish, especially beyond sample sizes of 1000 or so. Even 
professionally conducted national polls rarely sample more 
than 1000 people. For your purposes, you may be quite happy 
with a sample of 300 or so people. You should rarely need 
more than about 500. 

You may be wondering why no mention has been made of the 
overall size of the group you're sampling from. Doesn't it 
make a difference if you're sampling from 100 students or 
25,000 alumni? No, it doesn't. 

A few observations are in order here: 

1. Depending on your response rate , you may need to 
contact many more people to get the number of responses you 
would like. For example, if you would like 500 responses and 
expect a 25% response rate, you will need to survey 2000 
people initially. 

2. If you want to study any subgroups (for example, 
just freshmen or just students who withdraw or just business 
majors) , either plan a large enough overall sample that these 
are sufficiently large or realize that these subgroups will 
have a larger sample error. 

3. Note the size of your sample error when you report 
and interpret your results. Chapter VI (Analyzing tha Survey 
Data) will explain how to calculate it. 



HOW WILL YOU SELECT YOUR SAMPLE? 

For your survey results to be meaningful, you will want 
your sample to be representative of the entire group you are 
taking your sample from. Ideally, the best way to make sure 
a sample is representative is to choose it "randomly". In a 
random sample, every single person you'd like to survey has 
exactly the riame chance of being drawn. The concept of 
randomness is what makes sweepstakes fair (everyone who 
enters has an equal chance of winning the car or the trip to 
Europe) and it's what makes a survey both fair and accurate. 
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It lets you assume that your sample will closely resemble the 
entire group you are selecting from. The moment you stop 
being perfectly random in picking your sample, your sample 
error increases. 

S imple random samples can be generated in a number of 
ways. The easiest way is by computer: if the group you want 
to sample from (say, your undergraduate student body) is on a 
computer file, check if the computer can select a random 
sample for you. 

If your computer doesn't have this capability, you can 
generate a random sample with the simple and reasonable 
assumption that the last three or four digits of each person's 
social security number (or other identification number or 
heme telephone number) are randomly assigned. Let's say you 
would like to sample 600 students out of 5000 at yovr school: 

1. Social security numbers (or telephone numbers) end 
in 10,000 possible four-digit combinations (0000 through 
9999) . Since you would like to sample 12% of your students 
(600 out of 5000) , you would like to choose 12% or 1200 of 
the possible four-digit combinations. 

2. Using a random numbers table or by asking a few col- 
leagues for digits, choose a starting number — say, 3892. 

3. Your random sample will consist of all students 
whose social security numbers end anywhere from 3892 through 
the next 1200 numbers, up to 5092. If your computer cannot 
list these people for you, a student or clerk can go through 
a listing and identify the sampled students with a highlight- 
ing pen. 

There are other ways to obtain random samples: 

Systematic random samples involve taking every tenth, 
50th, or whatever person from a list. While this approach is 
simple, you may run the risk of emitting key cohorts from 
your sample. Let's say you are sampling 200 names from a 
1000-page phone book and you decide to choose the first name 
or every fifth page (page 5, page 10, etc.) . But maybe in 
your town there is a strong German group, mostly active and 
opinionated about your school. There are therefore a lot of 
names in the phone book beginning with "Sch". They start 
witn the second name on page 75C and run through the last 
name on page 754. None of these people has even a chance of 
being sampled! 

Stratified random samples involve breaking your entire 
group into subgroups and then taking a random sample of each 
subgroup. This ensures that even small subgroups (e.g., 
minority groups, small majors) are represented. 
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Cluster random samples involve taking a random sample of 
groups or "clusters" and then surveying everyone in those 
clusters. For example, you might take a random sample of 
freshman English classes and survey everyone in those classes. 
Or you might take a random sample of dormitory floors and 
survey everyone on those floors. Cluster samples are partic- 
ularly useful when you plan to distribute a questionnaire in 
person. 

Ail random sampling techniques assume that you have a 
complete and accurate list of all the people you wish to 
select your sample from. In reality, this is rarely the 
case. Alumni files, for example, are notoriously incomplete. 
Even rosters of currently enrolled students always include a 
few students with missing, incomplete, or outdated addresses. 
Surveys administered to a class in person will miss absent 
students . 

This problem becomes particularly serious when you are 
trying to survey the public at large, as you might in a 
community needs assessment. There is usually no complete, 
accurate list available of the people you wish to choose your 
sample from. Many people rely on telephone directories, 
which omit people with unlisted numbers, people who have 
recently moved, and any adults in a household who do not have 
a listing in their names (depending on the policy of the 
telephone company in your area, this could include many 
married women, who might be an important target ^or you) . 
Voter registration rolls have similar limitations. There is 
no way around this problem, except to recognize it as you 
plan your survey, analyze the findings, and report your 
results . 

There are some instances when a non-random sample is 
appropriate. Sometimes the list of the group you wish to 
sample from is so incomplete that even selecting randomly 
from it would give you a biased sample. Sometimes the list 
is not easily accessible (for example, an alumni list that 
has not been computerized) . Sometimes it takes too long to 
obtain the list (perhaps there 1 s a long waiting list for 
computer services) . Non-random samples are appropriate in 
these instances. Non-random samples can be valid, unbiased 
samples, but the burden of proof is on you to demonstrate 
that they are. 

There are two basic types of non-random samples. A 
judgement sample is carefully chosen so that, in your best 
judgement, it is representative of the entire group. 
National "test markets" are chosen this way. You might us^ 
judgement sampling to select a panel. A convenience sample 
is just that — a sample chosen because it was easily accessi- 
ble. If you ask a few faculty you're friendly with if you 
can borrow their classes for ten minutes to administer a 
survey, you're choosing a convenience sample. 
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WHAT SURVEY DESIGN SHOULD YOU USE? 



Much of the research in higher education consists of 
conducting one survey of one sample of people and describing 
(or perhaps explaining or exploring relationships among) the 
results. Before you decide to do this yourself, consider 
other possibilities. Your choice will depend on the purposes 
of the study and time and monetary constraints. 

One possibility is to survey more than one group and 
compare results among groups (comparing, for example, stu- 
dents who graduate with students who fail to graduate, bus - 
iness majors with education majors, men with women, freshmen 
with sophomores, last year's graduates with graduates of ten 
years ago) . 

Another possibility is to examine cnanges in one or more 
groups over a period of time. This is called a longitudinal 
survey. You might want to survey freshmen and then survey 
the same students four years later to assess changes in 
attitudes or goals. 

If you think a more complex design might be best for 
you, consult a faculty member from a social sciences, psy- 
chology, or education department for advice on experimental 
design . 



SHOULD THE SURVEY BE ANONYMOUS? 

Should responses be identifiable? Since many people may 
be afraid to share their views other than anonymously, this 
is not a simple decision to make. It depends on four things: 

The nature of your questions. The presumed advantage of 
an anonymous questionnaire is that it will increase respondents' 
chances of answering your survey and of answering it honestly . 
How important this is to you depends on the nature of the 
questions you are asking. A questionnaire dealing with sexual 
habits or political preferences may need anonymity much more 
than one asking about gardening habits. 

The people you are surveying. Will they be sympathetic 
with the goals of your survey or suspicious of your motives? 
With a group of well-educated people familiar with the nature 
of your work, you may not need an anonymous questionnaire to 
get the response rate you desire. 

How much factual information you need. If you want to 
break down your responses by sex, age, socioeconomic status, 
or other variables and you have access to such information 



outside the questionnaire, you may find a questionnaire that 
identifies the respondent is preferable for two reasons. 
First, you won't have to ask for such factual data on your 
questionnaire, shortening it and thereby increasing your 
response rate. Second, by collecting your factual data from 
an original source rather than asking for it to be self- 
reported, you increase its accuracy. Some people may not 
remember their high school rank or annual income, for example, 
while others may choose not to respond to such questions. 

The importance of follow-up mailings. With a mailed, 
anonymous survey, you lose any chance of following up with a 
second mailing to those who don't respond the first time. 
Since follow-up mailings can account for half of your total 
responses, this may be a serious problem. On the other hand, 
if you're handing out and collecting a survey of a "trapped 
audience", this is an unimportant consideration. 

The disadvantages of anonymous surveys frequently 
outweigh their advantages, especially when they are mailed. 
You will find that professionally conducted surveys are 
usually identified in some way. Try to make your question- 
naires as anonymous as possible by number-coding each rather 
than identifying it by name. Do everything you can in every 
other way to motivate the respondent to answer and make up 
for any diminished response rate. 



HOW MUCH WILL IT COST? 

Few of us have an open-ended budget. The rest of us 
need to consider how extensive a project we can undertake 
given our funds. In planning a budget for a survey research 
project , consider the following possible expenses : 

• Personnel time, including your time, your staff's time, 
and the cost of any special help (e.g., student workers, 
graduate assistants) brought in specifically for this 
project. 

• Printing the questionnaire, cover letter, return enve- 
lope, and outer envelope in sufficient quantity to cover 
follow-ups as well as the initial mailing. 

• Postage for the initial and follow-up mailings. 

• Incentives (e.g., a pencil, a quarter, a drawing for a 
gift certificate) . These are discussed in Chapter IV 
(Conducting the Survey) . 

• Follow-up costs other than printing and postage, includ- 
ing telephone calls and certified mailings. 
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• Data processing costs* 

• Duplicating and distributing the final report. 



WHEN WILL YOU GET IT ALL DONE? 

The final step in the planning process is to determine 
when the information will be needed for decisions and to work 
out a time line ensuring that everything will be done by then. 
If your report is not available when decisions must be made, 
all the time and energy put into it will have gone for naught. 

Make a list of all the things you will need to do to 
complete the project. For each item, note (1) when it must 
be completed, (2) how long it will take to do, and (3) when 
it must be started. Note which dates are flexible. You may 
need to trim your survey if you cannot complete it by your 
deadline . 

Appendix 1 gives an example of a time line form for a 
survey research project. 
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Chapter II. 




FORMATS FOR SURVEY QUESTIONS 



I 



Once a survey has been planned, the first step in 
developing the questionnaire is to construct questionnaire 
items that will get you the information you want. 

This chapter will describe the types of questionnaire 
formats available. The next chapter will discuss drafting 
individual items and integrating them into a questionnaire. 



YES/NO QUESTIONS 

Yes/no questions are questions with only two possible 
answers : 

Are you registered to vote? 

Do you support the legalization of marijuana? 
Are you a full-time college student? 

The two possible answers are provided along with a 
space to indicate the chosen answer. 

Advantages . Yes/no questions lend themselves to a wide 
variety of simple rnd complex data analyses. Responses can 
be summed into percents or treated as bivariate data and 
included in more complex analyses such as analysis of vari- 
ance or regression analysis . 

Disadvantages . Yes/no questions should be rarely used 
for two reasons. First, very few questions are purely yes 
or no; most have a grey area in between. Consider the 
sample questions above. A person may be registered to vote 
but in another state. He or she may support the legalization 
of marijuana in certain circumstances but not others. He or 
she may have been a full-time student for the past three 
years but is attending part-time this semester. 

Second, yes /no questions are not very interesting, 
either to the researcher or to the respondent. All they 
will tell you is how many people said yes and how many said 
no. They will not tell you why. And they may be boring 
enough to the respondent to make him or her throw the whole 
questionnaire into the circular file. 

To salvage a yes/no question - t you can make it more 
interesting by making it part of a "contingency question" 
such as this one: 

Are you attending college full-time or part-time 
this fall? 

Full-time. Go to Question 9. 

Part-time: Please rate your satisfaction with 

the following: 

21 
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Contingency questions have disadvantages, however, as 
discussed in the next chapter. 



MULTIPLE CHOICE QUESTIONS 

Multiple choice or fixed choice questions consist of a 
question or "stem" followed by a set of answers that the 
respondent chooses from: 

What is your one main source of financial support 
while attending college? 

My own earnings and savings 

My parents 

Financial aid and loans 

Other (please specify: ) 

Advantages . Multiple choice items are widely used, 
principally because they are quickly and easily answered and 
the responses are generally easy to tally. They are partic- 
ularly good for collecting factual information. 

Disadvantages. The chief difficulty in writing multiple 
choice questions is making sure you have included all possi- 
ble answers. If you're not sure you've mentioned every possi- 
bility, you can use an "Other — please specify" response, but 
don't use it to salvage a poorly written question. Pretest 
the item as an open ended question and use the most frequent 
answers as your responses. 

Another problem v;ith multiple choice questions is that 
respondents will often check more than one answer, even if 
you don't want them to. Your coding, analysis, and inter- 
pretation will be a lot harder if you allow multiple responses. 
Try to make clear in your directions that only the one best 
answer should be marked. If a few respondents still persist 
in marking two or more answers, choose one "randomly and dis- 
card the rest as you edit your responses. 

A third limitation of multiple choice questions is that 
the data they yield are usually categorical or ordered. 
This can seriously limit your data analysis options. One 
way to get around this is to code responses to a multiple 
choice question as a series of bivariate (yes/no) responses. 
Responses to this question, for example, 

Where do you live? 

In a campus dorm 

At home 

Off campus in a rented house or apartment 

could be coded as responses to three yes/no questions: Do 
you live in a campus dorm? Do you live at home? Do you 
live off campus? A respondent living off campus would be 
coded No-No-Yes. 
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Tips on writing. Here are some suggestions for writing 
multiple choice questions: 

1. The stem should ask a complete question, even if it 

is phrased as an incomplete statement. The respondent shouldn't 
have to read the responses to determine the question. 

2. Limit the number of possible choices to five or six. 
It's hard for most people to keep more than that many in 
mind . 

3. Your choices should all be mutually exclusive (i.e., 
clearly distinct and separate from one another) . 

4. If there is a dominant response that you're not 
interested in, include it in the stem (for example, "Aside 
from housing, what . . . ?" ) . 

5. Order the responses logically — numerically if the 
answers are numbers, for example. (If your set of responses 
is a list of numbers, most people will pick from the middle 
of the list.) 

6. "Stack 11 your responses vertically , with the boxes 
or spaces to be checked in a column on the left. This saves 
confusion and searching for the right space to mark. 



RANKINGS 

In a ranking item, the respondent is asked to number 
all responses according to specified criteria: 

Please rank each category below to indicate how 
important it was to you as a source of financing, 
Rank the most important source 11 1", the second 
most important source "2", etc. 

Full and/or part time work 

Savings 

Parental aid 

Grants (e.g., Pell) 

Loans 

Scholarships 

Veterans' benefits 

Social Security benefits 
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Advantages . The primary advantage of rankings is that 
many respondents can easily understand them. 

Disadvantages . Rankings have three serious weaknesses. 
First, they can be tedious to complete. Often a respondent 
must reread the list over and over as he or she is ranking 
it. 
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Second, they often incorrectly assume that respondents 
feel differently about every item on the list; this is often 
not true. 



Third, rankings are difficult to statistically analyze 
and interpret. They yield only ordered data which seriously 
limit your data analysis choices. You will never know if 
respondents felt their #1 and #2 choices were close together 
or far apart. This makes interpretation of your findings 
difficult. 

Tips on writing. The key to writing a good ranking 
item is to make sure your directions are complete and that 
the criteria for the ranking are clearly specified. Don't 
just say, for example, "Rank the following giving a 1 to the 
best and a 10 to the worst." Define in what respect you 
mean "best". Other tips: keep the list of items to be 
ranked no longer than ten items, all on one page, and make 
clear whether you want all irems ranked or just, say, the 
top three. 



LIKERT RATING SCALES 

Likert scales are probably the best-known rating scales. 
They are usually characterized by the headings "Strongly 
Agree/ 1 "Agree," "Disagree," and "Strongly Disagree," although 
the "Strongly Agree" and "Strongly Disagree" headings are 
optional . 

Strongly Not Strongly 

The instructor. . . Agree Agree Sure Disagree Disagree 

Spoke clearly. 

Graded fairly. 

Discouraged questions. ~~ 



Advantages . Many people are familiar with Likert scales 
and therefore find them easy to complete. Like other rating 
scales, Likert scales are efficient (a great deal of infor- 
mation can be provided quickly and compactly) and permit 
comparisons among answers within the scale. 

Likert scales yield ordered but not interval data. Many 
interval data analyses are sufficiently "robust," however, 
to be used with Likert data. This opens a world of possible 
statistical analyses . 

Disadvantages . Likert scales are most successfully used 
to measure attitudes or opinions rather than factual infor- 
mation. Novice researchers are often tempted to construct 
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an entire questionnaire of Likert items to facilitate data 
analysis. Don't do this if it means forcing some items into 
an inappropriate, awkward , or unclear format. 

It is often difficult to write unambiguous items for 
Likert scales. Think of the very different people who might 
agree with "I seldom attend parties 11 or disagree with "My 
math course requires about as much time and effort as other 
courses . " 

Tips on writing. Keep at least some of your items 
negative or presenting an opposing view. This will avoid a 
"yeasayer /naysayer effect 11 , in which someone with generally 
positive feelings toward the subject checks all the "Strongly 
Agree" responses without reading the individual items. 

Use of a "Not Sure", "Undecided", or "Don't Know" 
column is optional, depending on whether you want to force 
an opinion. If you decide to leave it out and force an 
opinion, you may be forcing a lie from someone unfamiliar 
with the issue. But if you include it, some people will use 
it as a way out for opinions they prefer not to take the 
time to think out. And you won't be able to tell (without 
further questions) whether an "Undecided" respondent is 
truly that or really "Uninformed". The decision depends on 
your questions and th3 people you are surveying. 



SEMANTIC DIFFERENTIAL RATING SCALES 

Another special kind of rating scale is the semantic 
differential. It consists of antonyms with either five or 
seven spaces in between. The respondent marks the space 
that best reflects his or her opinion: 



Advantages . Semantic aifferential scales yield inter- 
val data that are usabJe with virtually any statistical 
analysis. Like the Likert: scale, they are an efficiv. way 
to collect a great deal of information and they maK". ..'j>mpar- 
isons among answers straightforward. 

Disadvantages . I c is often difficult to give cerise 
written directions f'^r semantic differentials, especially to 
respondents unfamiliar with rating scales. 

Semantic differential items can only be ased whan you 



The library's services are: 



Useful 
Slow 
For Faculty 



Useless 
Fast 

For Students 



have obvious antonyms for the subject at hand, and they 
easily slip into trivia. One might be tempted to ask, for 
example, if the gym facilities are useful/useless, ade- 
quate/inadequate, well designed/poorly designed, and 
convenient/ inconvenient, when any one of these pairs might 
be adequate. 

Tips on writing. As with Likert scales, avoid a 
"yeasayer/naysayer effect" by putting some negative words or 
phrases on the left side as well as the right. Five spaces 
between antonyms are usually adequate for most purposes; 
people cannot dir cinguish well among more than seven. 



ECOSYSTEM RATING SCALES 

Webster [ s defines an ecosystem as "the complex of a 
community and its environment functioning as an ecological 
unit in nature." An ecosystem rating scale (also known as 
goal attainment scaling or GAS) asks for two ratings, with 
the second rating giving information on the environment in 
which the first rating was made. The second rating thus 
facilitates interpretation of che first by helping us 
understand the "complex" of the initial rating and its 
environment. 

Ecosystems are used in evaluation studies to measure 
the attainment of goals and objectives. They usually 
consist of a set of goal statements. The respondent is 
asked to rate each statement (usually on a 1 to 5 scale) 
twice ; first on the growth obtaine d and second on the 
growth desired ; 

Growth Growth 

obtained desired 

An appreciation of the limitations 

of our current knowledge. 

An understanding of the nature of ~~ ~ ~ 

science and scientific research. 

Awareness of the value of pursuing " — 

lifelong education. 



Responses in the second column greatly improve inter- 
pretation cf the first. If a statement receives a low 
"growth obtained" rating, for example, the "growth desired" 
rating will tell you whether you indeed have a problem. 

The ecosystem concept can be used to help interpret 
many other types of ratings or evaluations. The first 
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rating might ask for the respondent f ~ satisfaction with a 
service, for example. rp he second rating could then ask for 
the relative importance or value of that service, how 
frequently the respondent uses the service, or how much 
improvement the respondent has seen in the service over the 
last year. 

Advantages . Because of its dual-rating approach, an 
ecosystem is probably one of the best means available to 
measure the attainment of goals and objectives. It is 
efficient and yields interval data to facilitate data 
analysis. Its second rating greatly facilitates interpre- 
tation of your findings and makes comparisons easy. 

Disadvantages . The chief disadvantage of an ecosystem 
is that it is difficult to provide concise, clear writt n 
directions for it. It may be difficult, frustrating, or 
time consuming for some people to complete. 

Tips on writing. The trick to writing an ecosystem is 
to have good, clear directions. Describe the meaning of 
each l-to-5 rating number (e.g., excellent, good, etc.). 



OTHER RATING SCALES 

You can make up your own rating scale. Many people use 
the following: 

Excellent, good, fair, poor 
Frequently , sometimes , never 
Approve , undecided, disapprove 
Better , the same , worse 

The letter grades A, B, C, D, and E are an excellent 
choice when asking for evaluations. They are universally 
understood and yield interval data. They should be accom- 
panied by a definition of each grade, since some people 
consider C average while others consider it below average. 

Advantages . Any rating scales in a grid format give 
you a tremendous amount of information quickly and easily 
and let you compare responses to various questions. Making 
up your own rating scale is the appropriate choice when none 
of the question formats described in this chapter: appears to 
fit your needs. While rating scale data are often ranked, 
many interval data analyses are sufficiently "robust" to be 
used with this type of data. 

Disadvantages . When you depart from accepted formats, 
you run the increased risk that respondents may be confused 
by an unfamiliar rating scale. It is even more important 
than usual that you pilot test your questionnaire carefully 
before administering it . 
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Tips on writing. The general principles used to 
construct other rating scales apply to your own as well. 

Whenever you use a rating scale with a grid format, 
label each column with names, not just numbers. Don't ask, 
for example, for a rating of 1 through 5, with 1 being 
"most" and 5 being "least." People will have different 
concepts of 2, 3, and 4 unless you spell them out. Do you 
want 3 to mean "usual" or "expected average," for example? 

Many researchers number or letter-code each column and 
explain in the directions what each code means. This can be 
often confusing and frustrating to the respondent . 

You must be careful when constructing any rating scale 
to mix positive and negative statements to avoid a possible 
"yeasayer /naysayer " effect . 

Don't use more than four or five columns if possible — 
certainly no more than seven. People tend to shy away from 
extreme categories and stay in the middle. 



OPEN-ENDED QUESTIONS 

Open-ended or free-response questions leave a blank 
space where the respondent compos^ his or her own answer: 

Vvhat was the one best part of this program? 



Sentence completion items are a special kind of open- 
ended question: 

More than anything else, I would advise new 
freshmen to 



Advantages . There are four occasions when open-ended 
items may be best: 

1. When many answers (more than six or seven) are 
possible, or when a multiple-choice question might overlook 
some important responses . 

2. When you are asking for occupation. 

3. When a multiple-choice item might bias responses by 
steering respondents in a particular direction. 

4. When your question is interesting enough that 
people will want to answer it. People enjoy being asked, 
"How would you improve...?", for example. 
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There are a few other advantages to open-ended 
questions. They are useful in pilot studies for drafting 
more specific questions for the final study. And a few 
quotes from open-ended questions can add "sparkle" and 
credibility to your report and help readers uncomfortable 
with numbers understand your results. 

Disadvantages . Open-ended questions are not very 
popular with respondents, since they make them work harder 
and lengthen the time required to complete the questionnaire. 
Very often the respondent will leave them blank, along with 
the rest of the questionnaire! 

They're not very popular with researchers, either, 
since the answers are generally difficult to read, cate- 
gorize, process, and interpret. The data must be analyzed 
subjectively, not objectively. 

Tips on writing. The amount of space allotted for an 
open-ended item should give a clr^ to the expected response 
length. Keep it as short as possible, and rule the space to 
make answers easier to write and read. 

Sentence completion items will be most successful if 
they ask for a response of just a word or two and if the 
blank is at the end of the sentence. 

The major problem with open-ended questions is figuring 
out how to analyze the answers. Coding responses to open- 
ended questions is discussed in Chapter V (Processing the 
Survey Results) . 
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Chapter III. 



DEVELOPING THE SURVEY INSTRUMENT 



Now that we have discussed planning your survey and the 
kind of questionnaire items you might employ -~u"re ready 
to draft the questionnaire. This chapter v ; discuss 
drafting individual items and assembling th^ into a ques- 
tionnaire. But, first, let's talk about a question that's 
probably uppermost on your mind: 

HOW CAN YOU MAXIMIZE YOUR RESPONSE RATE? 

Four factorc will probably have the most effect on your 
response rate: 

1. The topic of your survey. A survey asking for 
simple, non-threatening facts will probably get a higher 
return rate than one asking for opinions on a sensitive 
issue . 

2. The people you are surveying. You ■ 11 probably get 
a higher response rate if your subjects are sympathetic with 
your project and find it interesting and relevant than if 
they find it uninteresting or valueless . 

3. How considerate you are of your respondents. 
Recognize that you have no right to expect strangers (or 
even acquaintances) to go to all the trouble of filling out 
a form and that they are doing you a great favor when they 
do. If you show your appreciation by doing all you can to 
minimize your respondent's trouble and make his or her job 
as easy as possible, he or she will be much more likely -co 
respond to your survey and give you valid information. 

4. How professional and important the study appears. 
If you appear professional and the study appears important, 
you will make your respondent's contribution seem much more 
worthwhile . 

There is not much you can do to a 1 er the first two 
factors. The only way to handle respondent concerns about a 
survey on a sensitive topic is to emphasize the confidential- 
ity and professionalism of your survey and to include the 
name and telephone number of someone they can contact with 
questions. The only way to handle unenthused subjects is 
with a convincing cover letter. 

But you can do many things about the last two factors — 
BEING PROFESSIONAL AND CONSILERATE OF YOUR RESPONDENTS — and 
therefore maximize your response rate. 
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To find out how considerate you are of your respondents 
(and how well you are maximizing your response rate) , jsk 
yourself t'xe following: 

1. Hov; long is your questionnaire? The shorter it is, 
the more considerate you are cf your respondent's time. 

2. How long does it appear to be? A cluttered page of 
minuscule type looks long and complicated. 

3. How clear are the questions? Mus the respondent 
spend time trying to figure out what you really mean? 

4. How clear are the directions and layout? Can the 
respondent move easily from one question to the next? 

5. Was the questionnaire pilot tested? If it wasn't, 
you can't be sure your questions and directions are clear 
enough. 

6. Which questions are first? Last? The long, 
complicated, and boring questions should be at the end, 
where your respondent will see that the questionnaire won't 
take much longer to complete. 

7 . Is a stamped, addressed return envelope enclosed? 
If it isn't, you're telling your respondents ycu really 
don't care whether you hear them o*- not. 

8. When is the survey mailed? Will it arrive at a 
time when your respondent is busy with other matters, such 
as before a weekend or holiday? 

9. Do you offer a summary of the findings, so your 
respondent can see the impact of his or her efforts? 

10. Have you enumerated all possible reasons for people 
in your sample not to respond to your questionnaire? Have 
you done all you can to overcome those obstacles? 

To find out how professional you appear to your res- 
pondents (and how well you are further maximizing your 
response rate) , ask yourself the following: 

1. Does your questionnaire ask interesting, important- 
sounding questions? 

2. Is it carefully laid out and grammatically flawless? 
3* Does it have a title and sponsor? 

4. Is it well-reproduced on quality paper? 



5. Is the cover letter convincing of the importance of 
the study and the value of each individual response? 

6. Will a follow-up mailing be conducted? 

This chapter and those following will address each of 
these points in more detail. 

A great deal of research has been done on other factors 
affecting response rate — things like item type, print size, 
paper color, etc. — often with conflicting results. Many 
studies support the contention presented here that surveys 
that are professional and considerate of respondents have 
higher response rates. In many other research studies, the 
effects studied appear to interact with the nature of the 
study and its subjects. 



WHAT GOES INTO A GOOD QUESTIONNAIRE ITEM? 

One of the most important ways to be considerate of 
your respondents is to write questions that are CLEARLY 
UNDERSTOOD. There are a number of ways you can make sure 
your question will be clear to your respondents: 

1. Keep it short. Short, straightforward items are 
easier to understand than compound or complex statements and 
therefore yield more accurate answers. They also mean a 
shorter questionnaire and therefore an increased response 
rate. Lengthy questions may cause the respondent to lose 
focus and may also cause fatigue and impatience. 

Here is an example of a question that could be made 
much shorter: 

Which of the following statements best describes 
your enrollment pattern at Briggs College? 

I was enrolled primarily as a full-time 

student . 

I was enrolled primarily as a part-time 

student. 

Rewrite each question several times, trying to simplify 
each sentence as much as possible. 

2. Keep it readable. Keep the vocabulary level low; 
go over each word in your item and make sure it is as simple 
as possible. Avoid the jargon of your field; don't use 
words like "attrition" or "articulation," for example. 
Here's an * em that's loaded with jargon: 
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Please indicate yes or no in each of the following 
cases where you use computing services. 

1 Data Base Management 

2 Word Processing 

3 Record Keeping 

4 Spreadsheet Analysis 

5 Data Analysis and/or Research 

6 Programming 

I learned through a pilot study that the following item 
also has jargon. 

Are you primarily interested in taking courses for 
credit, or would you prefer to attend non-credit 
courses^ 

Credit 

Non-credit 



"Credit 11 and "non-credit 11 may be incomprehensible to 
people who have rever gone to college. 

Avoid negative items. If you must have them, capital- 
ize and/or underline NQ1 , EXCEPT , etc. Beware of double 
negatives and other eas'ly confused phrases. 

3. Make sure it asks only one question. Rather than 
ask, "Should 18 year olds have the rights of an adult?", 
specify which rights you mean: voting, drinking, marriage, 
or whatever. A "double-barreled" item like the following 
asks several questions that might each be answered differ- 
ently: "Do you feel that computer literacy is, or should 
be, an objective or requirement of the college?" To avoid 
this problem, check your use of the words "and" and "or" and 
limit your questions to one adjective or adverb each. 

4. Keep it interesting. Don't let your items get 
pedestrian or monotonous; vary them as much as you can. The 
six items below would quickly frustrate almost any respondent 

How important is your need to locate information 
of the following types? 

Least Most 

important important 

Simple facts for course purposes 
Simple facts for research purposes 
Simple facts for general purposes 
Complex data for course purposes 
Complex data for research purposes 
Complex data for general purposes 
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Many people write questionnaires using the same kind of 
item format (e.g., a Likert scale) throughout. While this 
makes it very easy to compare responses to different items, 
it can also make the questionnaire monotonous and boring. 

5. Make all definitions, assumptions, and qualifiers 
clearly understood. Clarify terms that could be misunder- 
stood. A question like "Do you use the IBM mainframe 
computer?" can be interpreted many ways. One person could 
say that anyone who's ever looked at a printout is tech- 
nically a user; another could say that only those with 
"hands on" experience are users. Similarly, rather than ask 
for age, ask for year ot birth or age as of January 1. 
Terms like "value," "how much," "usually," "most," "quality," 
and "now" are full of ambiguities . 

When asking for an estimate of something, spell out the 
period of time you are interested in: last week? per day? 
When you ask for salary, similarly specify whether you want 
gross or net salary and whether you want salary rate per 
year, per month, biweekly, weekly, or per hour. In fact, 
specify the unit of measurement for all numerical responses. 
Do you want the student ! s course load expressed as number of 
courses or number of credits? Label the response blank with 
the appropriate unit of measurement to help clarify your 
question . 

When asking for a rating or comparison, make sure you 
give a clear point of reference. Spell out the criteria to 
be used. Do you want the staff rated on courtesy, cheerful- 
ness, helpfulness, or knowledge? Is "above average" in 
terms of actual or desired performance? In terms of em- 
ployees here or employees in general? 

If your question includes a qualifying phrase (e.g., 
"except for housing," "if you commute"), place it at the 
beginning of your stem. Your sentence structure may not be 
the best, but the qualifier won't be overlooked. 

Make sure your questions have no hidden assumptions 
written into them. How many assumptions can you find in 
this classic: "Have you stopped beating your wife?" 

When it comes to definitions, assumptions, and quali- 
fiers, consider the feelings of your respondent. Don't 
phrase definitions in such a way that he or she feels 
stupid. Make sure the respondent has all the information 
necessary to answer the question. Be careful of presuming, 
say, that the respondent has graduated from high school. 

6. Make your question as specific as possible. When 
you are asking a question with a numerical or frequency-type 
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response, consider just leaving a blank for the person to 
fill in the response rather than giving a series of multiple 
choice responses or an always/sometimes/never rating scale. 
Consider this question: 

How much do you usually spend on textbooks each 
semester? 

$ 0-19 

$20-39 

$40-59 

$60 or more 



This item is poor for several reasons. First, it re- 
quires the respondent to make a mental average of spending 
over several semesters. Second, it has five lines of 
reading (it's not a very long item but it could be shorter) . 
Third, the responses will be ranked data, not scaled. You 
will not be able to calculate the average spending on text- 
bcoks or perform many other statistical analyses. The 
question would be greatly improved if it were rewritten as 
follows : 

About how much did you spend on textbooks last 
fall? $ 



The only times it does pay to put a numerical answer 
into a multiple choice or rating scale format is with 
questions in which respondents may not remember the exact 
amount ("How many times have you called home in the past 
year?") and with sensitive questions like age or salary. In 
those cases, people will be more willing and able to indi- 
cate a general category than an exact answer. 

7. Consider making your question behavioral. It is 
often harder for people to express their feelings or 
opinions than to describe concrete actions. Opinion 
questions are also often harder to analyze and interpret 
than behavioral "vignette" questions, since respondents 1 
views come fra, different frames of reference. Student 
opinions of your advisement program, for example, may be 
based in part on the quality of students 1 high school 
advisement experiences or on stories they've heard about 
advisement at other colleges. 

Your questionnaire will therefore be less ambiguous and 
more reliable if you make as many items as possible concrete 
or "behavioral:" asking for a behavior rather than a feeling 
or opinion. For example, instead of asking, "Do you consider 
yourself a hard working student?," asK, "How many hours per 
day do you usually study?" or "How many hours did you study 
yesterday?" Try asking several questions and inferring an 
opinion rather than simply asking for the opinion. 



Don't try to make your questions too specific, however. 
A large number of "trivial" questions may cause confusion or 
a loss of interest and will lengthen your questionnaire 
considerably. 

8. Avoid loaded, leading, or sensitive questions. 
Many people are comfortable with the status quo and reluc- 
tant to admit any inferiority or wrongdoing. They may lie 
if you ask them whether they smoke, watch TV, go to art 
museums, or vote. Try not to phrase questions in a psy- 
chologically threatening way. Rather than ask a res- 
pondent does something good or bad (e.g., vote, cheat on 
tests) , ask how often he or she does it. It also helps to 
ask about very specific behaviors. If you ask, "Did you 
glance at your neighbor's paper during the last math test?," 
some respondents may not even realize you are asking about 
cheating behavior. Others may recognize your intent but 
still feel they can answer your question honestly without 
labeling themselves "cheaters." 

Questions worded in the first person ("Do you cheat?") 
will be answered differently from those in the third person 
("Do your friends cheat?") . Some respondents may find 
third-person questions easier to answer honestly. 

9. Avoid biased questions. While no one writes these 
kinds of questions on purpose, it's easy to write them inad- 
vertently. Here are two examples: 

Do you agree that computer literacy should be an 
objective or requirement of the college? 

Should harsher penalties be enacted for crimes 
committed against the elderly? 

The first question is subtly biased because of the word 
"agree". Respondents generally find it easier and less 
threatening to agree than disagree. The second question is 
biased because of insufficient information on both sides of 
the issue; as the question now stands, who would disagree 
with it? 

One of the best tests for biased questions is to imagine 
you're writing questions from the opposite point of view. 
For example, if you're trying to collect data to support tne 
need for expanded counseling services, imagine you're trying 
to cut back on them. Would you still ask the same questions 
and phrase them the same way? 

Another way to avoid bias is to ask about both the pros 

and cons of a sensitive issue. Don't ask the respondent to 

make criticisms without giving him or her a chance to praise 
as well. 
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10, Make thq item easy and fast to answer. The res- 
pondent should easily understand how to decide on an answer 
and how to record it. A simple answer format requiring only 
a few checks is better than a complicated scoring key. 

11, Design the item to yield data that are meaningful 
and easily interpreted. Don't ask unimportant questions. 
Review each one to see how it contributes to the objectives 
of your study. You should be able to explain why you are 
asking each question. 



SHOULD YOU BE CONCERNED ABOUT THE VALIDITY AND RELIABILITY 
OF YOUR ITEMS? 

This depends on the subject and purpose of your study. 
A simple, straightforward "one shot" study on an uncontro- 
versial subject , whose results are only for general infor- 
mation and not for a specific decision, probably doesn't 
need much evidence of reliability or validity. You should 
consider collecting evidence of reliability or validity, 
however, if your study has any of the following 
characteristics : 

• Your findings may lead to major changes. 

• The study is about a controversial subject and the 
findings may be hotly disputed. 

• The study is about a sensitive issue that respondents 
may not answer truthfully on. 

• There is disagreement about the definitions or 
assumptions you are building into your study (e.g=, Do 
all students need an academic advisor? What 
constitutes advisement?) . 

• The study is part of a substantial, expensive, perhaps 
longitudinal research project. 



HOW IS RELIABILITY ESTABLISHED? 

A reliable questionnaire is a questionnaire that elicits 
consistent responses. This usually means that responses to 
similar questions within a questionnaire should be similar. 
You can measure this kind of reliability (called internal 
consistency) by comparing answers to two or more similar 
questions. You would expect at least a moderate correlation, 
for example, among a group of items all asking about math 
anxiety. You can identify inconsistent responses by putting 
two very similar questions in different parts of the question- 
naire and seeing if the responses are similar. 
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The problem with measuring the internal consistency of 
a questionnaire is that you must ask at least two questions 
on the same subject, and to keep the questionnaire as short 
as possible we often ask only one question on each topic. 

In some instances yoa may also be concerned with re- 
sponses remaining consistent over a period of time. This is 
only a rare concern, however, since most questionnaires dea3 
with opinions or other information that is expected to chang 
over time. Indeed, the purpose of a questionnaire study is 
often to collect information to help us facilitate change. 
If you decide you need evidence of reliability over time, 
you will need to administer an identical questionnaire at a 
later date to the same sample of people and correlate their 
responses to the two questionnaires. 



HOW IS VALIDITY ESTABLISHED? 

There are a number of definitions of validity. Perhaps 
the most important characteristic of a valid questionnaire 
is that it measures what you want it to . A survey designed 
to measure freshman satisfaction with the orientation pro- 
gram, for example, should measure just that. Validity 
further means that each respondent should interpret each 
question in the same way, which in turn means that your 
questions must be consistently clear. 

There are a number of ways you can make sure your 
questionnaire is valid: 

1. Examine the items. They should have an intuitive 
relationship to your study's topic and goals. 

2. Pilot test your questionnaire (see Chapter IV) to 
make sure it is clear and easily understood. 

3. Keep your items free from bias and ambiguities, 
using the tips given earlier in this chapter. 

4. If you are asking about a topic that respondents 
may not answer truthfully on, include a few items solely to 
check the validity of responses. You can identify liars, 
for example, by asking a bogus or nonsense question, such as 
whether the respondent belongs to a nonexistent group. 

5. Compare your findings with independent factual data 
or with findings from similar studies. 

Another approach to validity recognizes that while you 
can never ask all the questions you would like to, you 
should make sure you ask an adequate sample of all possible 
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questions. This kind of validity can be evaluated only by 
examining the items carefully and sharing them with other 
people knowledgeable about the subject. If you are doing an 
attrition study, for example, you will want to review the 
literature and talk with people on campus for ideas on why 
students drop out and then include as many of these ideas as 
possible in your questionnaire. 

A third approach to validity is important when you are 
using survey responses to predict a behavior, such as drug 
use or dropping out of school. If you wanted to design a 
valid questionnaire to identify students who have the po- 
tential to turn to drugs, you might administer your question- 
naire to a group of drug-free students and a group known to 
have used drugs. If the questionnaire is vaAid, you will 
get very different responses from the two groups. 



HOW SHOULD THE ITEMS BE ORDERED IN THE QUESTIONNAIRE? 

You will find this task easier if you write each question 
on a separate card or slip of paper. 

The first questions should be chosen with care. They 
should be questions that will "hook" the respondent into 
answering the survey and thus increase your response rate. 
Your first questions should have the following character- 
istics: 

1. They should be intriguing , perhaps asking for an 
opinion on an interesting topic or for some interesting 
information on the iespondent's background. Demographic 
questions (e.g., on sex, age, major) should not go first. 

2. They should be easy to answer . They should not 
require a lot of deep thought anc? should be answerable with 
a simple check mark. This is net the place to put compli- 
cated rating scales or open-ended questions . 

3. They should be general . Most people have an easier 
time moving from generalities to specifics than the other 
way around. Check if the first questions will influence 
later responses. If so, is this what you want? 

4. They should be somewhat impersonal . While you can 
ask immediately about general opinions or background, save 

the sensitive questions for later. "Favorable" questions—ones 
asking about accomplishments, for example — are good to put 
first . 



The remaining questions should follow a natural flow, 
both logically and psychologically. Try to keep questions 
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on a similar topic together, so the respondent can stay on 
one train of thought. Within each topic, try to begin with 
general questions and move into specifics. Gallup, for 
example, first asks respondents about general feelings on an 
issue and then asks about specific parts of the issue. 

Try also to group items of the same format together. 
This is especially important if you're using unique item 
types with complicated directions. 

Avoid contingency questions, those where if you check 
"yes" to one question, you then "GO TO" another set of 
questions. They're confusing and frustrating and therefore 
reduce response rates. It's probably easier to have every- 
one answer all questions and then delete non-applicable 
answers yourself during your data analysis. If you do have 
several questions that you'll want only a small group to 
answer, put them at the end of the questionnaire and tell 
everyone else to stop before reaching them. Or perhaps you 
can send this group a separate questionnaire. 

The last questions should be the ones youx respondents 
will be least enthused about answering. We'll hopo that by 
this they'll be "hooked" and won't mind answering these last 
few questions. If they do decide to skip them, at least you 
will have already gotten some data! There are four kinds of 
questions that should be as near the end as possible: 

1. Delicate, intimate, or sensitive questions . These 
include questions on age, salary, personal habits and pre- 
ferences such as how often someone drinks, and opinions on 
sensitive subjects like abortion. 

2. Complex questions that take a long time to read or 
a lot of thought to answer. 

3. Open-ended questions since these take a long time 
to answer. 

4 . Boring questions, including demographic items. 
Make sure you really need each piece of demographic infor- 
mation. Your respondents will find them irritants at best 
and offensive at worst. Try to make clsar to your respond- 
ent why you need this information. 



WHAT ELSE GOES INTO A QUESTIONNAIRE? 

Beside your questions, your questionnaire needs a 
title, directions, and a closing. 

Title and sponsor. You'd be surprised how often these 
are neglected! They are important tools in convincing your 
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respondent of the professionalism of your study and there- 
fore in increasing your response rate. The title lets tne 
respondent know what is to come and is an important motivat- 
ing device. ake it intriguing enough to pique the res- 
pondent's interest and help convince him or her to respond. 
Including the study's spon- s ("Office of Institutional 
Research, Atlantic State College") gives the questionnaire 
credence and legitimacy . 

Many people leave the title and sponsor off the question- 
naire because they are identified elsewhere — either in a 
cover letter or by the person administering the survey. 
Even in these instances, however, it's important to include 
chese things right on the questionnaire. They add a pro- 
fessional touch that will help convince respondents to take 
your questionnaire seriously. 

Directions should make the respondent want to answer 
truthfully. They should be simply phrased and crystal clear, 
telling the respondent how to answer each question and how 
to record his or her answers. Make clear, for example, 
whether the respondent should pick the one best response or 
as many as apply. Can he or she elaborate on responses? If 
so, where? Use boldface, capitals, or italics to draw 
attention to the directions. 

Generally speaking, shorter directions are easier to 
understand : 

Confusing; Please mark the number which more 
accurately reflects your opinion concerning each 
question listed below. Circling number 1 would 
indicate excellent and complete agreement, circling 
number 2 would indicate satisfactory and general 
agreement, circling number 3 would indicate unsat- 
isfactory and general disagreement, and circling number 
4 would indicate poor and total disagreement. 

Better : For each statement below, check the box 
thai most accurately shows your agreement. (These 
directions would be followed by statements with four 
columns of boxes labeled ("Strongly Agree," "Agree," 
"Disagree," and "Strongly Disagree.") 

Directions are simpler when you use as few different 
response methods as possible. You may want to consider 
short introductions to each section to explain new response 
methods and help respondents "shift gears". 

Closing. Don't forget to say thank you! Also repeat 
directions on how to return the questionnaire. A box to 
check (and space for an address) if the respondent would 
like a copy of the results is a nice touch. 
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HOW LONG SHOULD A QUESTIONNAIRE BE? 

The shorter the questionnaire, the less formidable it 
looks and therefore the higher the response rate. (Up to a 
point: once a questionnaire is over ten pages long, whether 
it is ten, fifteen, or twenty pages long will not affect the 
response rate.) The ideal questionnaire is on a double 
postcard and the next best is on one piece of paper . 

If your questionnaire draft does not appear to fit onto 
one piece of paper, here are some ways to make it fit: 

1. Shorten it! Go over the entire questionnaire. 
Eliminate the questions that would be interesting but are 
really unessential. Try to eliminate all unnecessary words 
and phrases . 

2 . Reproduce a two page questionnaire onto the front 
and back of one piece of paper. 

3. If the questionnaire won't quite fit onto the front 
and back of an 8 1/2 by 11 page, use an 8 1/2 by 14 page. 

4. Break your questions into two or more question- 
naires, sending each to a randomly chosen fraction of your 
sample. Ask everyone the essential questions, but distri- 
bute the remaining questions among the questionnaires. 

5. Use small type. Professionally typeset your question- 
naire if you can. If you can't, perhaps ycu have the facili- 
ties to "shrink" your typewritten page so it looks smaller 
(but don't make it so small that it's hard to read!). At the 
very least, use elite type instead of pica. 

6. Leave off the cover page. All it does is warn the 
respondent that a mighty long questionnaire is underneath! 



WHAT SHOULD YOU KEEP IN MIND AS YOU HAVE THE QUESTIONNAIRE 
TYPED AND DUPLICATED? 

A carefully laid out, grammatically flawless question- 
naire looks shorter and more professional, is much easier to 
complete, and therefore yields a higher response rate and 
better quality data. Here are some suggestions: 

1. Cleanly format your questionnaire. Two columns 
make it look shorter and are easier to read. 

2. Use plenty of "white space." Use generous margins 
and spacing. If your questions fill only three quarters of 
a page, spread them out. 
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3. Small type makes your questionnaire look not only 
shorter but also more professional. 

4. "Stack" the responses to multiple choice items verti- 
cally. A respondent can find and mark the appropriate re- 
sponse more easily when the choices aze listed in a column 
rather than across one or two lines : 

Confusing: Yes No 



Better: Yes 
I No 

Responses in one column are also easier for you and 
your helpers to find and code. And columns of responses 
create white space that adds eye appeal. 

5. Make spaces for check marks with professionally 
drawn boxes, brackets such as [ ], parentheses such as ( ) , 

or simply a short line such as _ # Typed boxes made with 

slashes and underscores look terrible and hand drawn boxes 
even worse. 

6. If possible, avoid "op-scan" sheets or those data 
entry codes that we're seeing on more and more question- 
naires. They look impersonal, giving the impression that no 
one will look at any one person's response or care what he 
or she has to say. They also make the questionnaire more 
confusing or difficult to complete. Use data entry codes 
only if: 

• You are sending out thousands of questionnaires. 

• You are asking for only very impersonal, factual 
information. 

• The codes may help convince your respondents of the 
professionalism of your survey. 

• Your data entry clerks cannot deal with a form without 
these codes (this is very unlikely) . 

• You or an assistant cannot code them yourself after 
they are returned. 

Pre-coded questionnaires are discussed further in 
Chapter V (Processing the Survey Results) . 

7. Proofread your questionnaire carefully for grammar 
and spelling. A single error will detract from the pro- 
fessional appearance of your questionnaire. Ask some lay- 
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people outside your field to read your questionnaire and let 
you know of any directions, items, or words they find hard 
to understand. 

8. Reproduce your questionnaire by the best means 
possible. Offset printing looks best and is not too ex- 
pensive. Dittos, photocopies, and mimeographs can be hard 
to read and don't look professional. 

9. Use quality paper. Subtly colored paper — nothing 
garish or too bright to be readable — is a nice touch if 
possible . 

Appendix 2 gives a few examples of questionnaires that 
have been used in institutional research surveys. 
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Chapter IV. 
CONDUCTING THE SURVEY 



Once you have prepared your questionnaire, ycu are 
ready to conduct the survey. In this chapter, we will 
discuss preparing a questionnaire package, conducting a 
pilot test, and administering the survey. We will also 
discuss what tasks can be delegated to others and ways you 
can use a microcomputer or word processor to help you. 



WHAT IS A QUESTIONNAIRE "PACKAGE"? 

If you are mailing your questionnaire, you will need to 
prepare three additional items: a cover letter, an envelope 
to return the questionnaire in, and the envelope that you 
will mail everything in. Each item, when properly prepared, 
can help motivate the respondent to answer; indeed, that's 
the whole purpose of the cover letter. 



WHAT SHOULD GO INTO A COVER LETTER? 

The cover letter speaks in your absence. It says all 
the things you would say in person to establish and maintain 
a rapport with the respondent and convince the respondent to 
decide to complete your questionnaire. It must do all this 
quickly and succinctly. 

There are several ways you can persuade the respondent 
to answer : 

1. Make the study look important. Explain the nature 
and purpose of the study and why you are asking the questions 
you have chosen. Show that the study addresses important, 
interesting questions and issues and explain how the results 
will be used to benefit someone. Stress that the respondent 
is part of a carefully selected sample and you need his or 
her response . 

To further impress the respondent with the importance 
of the study, encourage an immediate response. Date the 
letter and set a deadline. 

2. ft^peal to the respondent's self-interest. If at 
all possible, try to get a sense of your respondents' loyal- 
ties and priorities and show how your results will either 
benefit them directly Oi benefit some cause or issue they 
are concerned about. With adult groups, for example, you 
might be able to stress how your study will lead to saving 
taxpayers ' mon^y . 
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A more direct approach is to give the respondent a 
material incentive to answer. It may increase your response 
rate by giving the respondent a sense of obligation. Your 
incentive should be carefully chosen to have the most appeal 
to your particular group. 

• Enclose a pencil, a quarter, a magazine, or some other 
small item of value. 

• Promise to make a charitable contribution if the question- 
naire is returned. 

• Offer free tickets to a college event if the question- 
naire is returned. 

• Put all returned questionnaires into a drawing for some- 
thing of value (a $100 scholarship, a gift certificate 
from the college bookstore, a microcomputer) . 

You can also appeal to the respondent's self-interest 
by clarifying how easy his or her role is. Give the res- 
pondent a clear idea of how long it will take to complete 
the questionnaire (assuming it won't take more than a few 
minutes) . Note that a stamped, self-addressed envelope is 
enclosed . 

In case all else fails. . .threaten the respondent (in 
a nice way) . Tell him or her to expect a barrage of letters, 
phone calls, and what-have-you if the questionnaire is not 
returned ! 

3. Make the study look professional. Make the re- 
spondenc feel he or she is collaborating with professional, 
scientific researchers who are authorities on this issue. 
Have the letter immaculately typed on quality letterhead 
paper — it conveys an official sanction of the project. Have 
an important person sign the letter — preferably a name that 
respondents will recognize and react favorably to. For most 
surveys conducted by colleges and universities, the presi- 
dent should sign. If it's not important enough fcr him or 
her to sign, is it important enough to do at all? 

4. Address the issue of confidentiality. Regardless 
of whether your survey is anonymous, guarantee confidential- 
ity. If your survey is totally anonymous, make this clear. 
If it is not, explain why {at least explain the need for 
follow-up mailings) . Stress that you are interested only in 
aggregate responses and that the respondent's questionnaire 
will be separated from any personal identification. 
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5, Personalize the letter. We all hate impersonal 
form letters. You should therefore try to personalize the 
letter in some way unless the topic of your survey is so 
sensitive that respondents will want to feel as anonymous as 
possible . 

A personally-typod letter is the best way to personal- 
ize a cover letter. It may be feasible if you have access 
to word processing equipment (check with your admissions 
office) . If you must print your cover letter, you can add 
touches to it to make it less cold: 

• The name and telephone number of someo'ie to contact 
should they have questions about the study. 

• A signature in contrasting ink, preferably hand signed 
with a ball point pen so it looks hand signed. 

• Individually typed addresses and salutations. 

• A P.S., preferably handwritten. 

• An offer to send a copy of the results. 

• A handwritten follow-up note. 

Appendix 3 gives an example of a cover letter that 
incorporates many of these suggestions. 



DO YOU NEED A RETURN ENVELOPE? 

Absolutely! Remember the cardinal rules: Be pro- 
fessional and be considerate of your respondent. Asking a 
respondent to provide his or her own envelope or to fold a 
questionnaire in a certain fashion and find a stapler or 
piece of tape does neither of vhese — and accordingly 
generates a lower response rate. 

The return envelope snould be addressed and postage- 
paid. To do any less tells your respondent you really don't 
care whether you get his or her response or not. Some 
researchers have had better luck with a stamped envelope 
than a metered or business reply envelope. It should be 
legal-sized so the questionnaire doesn't need to be folded 
an extra time to make it fit. 

An alternative to enclosing a return envelope is to 
require the student to hand deliver his or her response as a 
prerequisite to receiving something of value. The most 
common use of this technique is with surveys of graduating 
seniors; students are required to hand in a completed survey 
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before receiving their cap and gown for the commencement 
ceremony. The disadvantage of this approach is similar to a 
disadvantage of in-person administrations (covered later in 
this chapter) : respondents may feel forced to do something 
they would rather not do. Some may falsify their answers as 
a way of rebelling against the process; others may rush 
through the survey at the last minute. Both adversely 
affect the validity of your results. 



HOW SHOULD THE QUESTIONNAIRE PACKAGE BE ASSEMBLED? 

The envelope your questionnaire, cover letter, and 
return envelope are delivered in conveys that important 
first impression. It should have a cleanly typed address or 
neatly aligned mailing label and ai official-looking return 
address . 

When you stuff the envelopes, first discreetly number 
the questionnaire on a back corner and enter the same number 
next to the respondent's name on your master mailing list 
(unless you are conducting a totally anonymous survey) . 
Fold the cover letter and questionnaire accordion-style and 
slip them in the envelope with the return envelope so that 
when the respondent opens the envelope, the first thing he 
or she sees is the top of the cover letter. 



WHY DOES THE QUESTIONNAIRE NEED A PILOT TEST? 

You probably think that, with all the work you've put 
in up to now, you've got to have a great questionnaire 
package, but you couldn't be more wrong! There's bound to 
be at least one question that's perfectly clear to you but 
confusing or misleading to respondents. Pilot testing is 
the most important step in a survey and the ono most often 
omitted by inexperienced researchers. 

Pilot tests can be very extensive or very simple. In a 
full-blown pilot study, the entire survey is conducted just 
like the "real" one, including designing the sample and 
collecting, processing, and analyzing the data. This lets 
you check not only the questionnaire package but the entire 
research process. Full pilot studies are most often done 
when a project is very expensive or important. 

The simplest pilot test is ^o hand out the question- 
naire to a few people similar to hose in your sample (for 
example, a few students if you are nirveying the student 
body) . Ask them not only to complete the questionnaire but 
also to tell you about any questions or directions they 
found unclear or ambiguous and how long it took them to 
complete the questionnaire. This gives you some feedback on 
your questionnaire but not on the effectiveness of the 
entire package. 
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Once your pilot test — simple, complex, or in between 
— is completed, go over the responses, looking for unex- 
pected or inconsistent answers, and revise your question- 
naire accordingly . 



WHEN SHOULD THE QUESTIONNAIRE PACKAGE BE MAILED? 

The questionnaire should arrive at a ^ime when the 
respondent is least likely to be busy and tempted to put it 
off. This generally means timing your mailing so the 
questionnaire arrives on a Tuesday, Wednesday, or Thursday. 
Avoid questionnaire arrivals on Friday or Saturday, during 
midterms or finals, or during holiday or vacation periods. 

You may want to consider an advance postcard, mailed 
four to seven days before the questionnaire. It lets 
respondents know the mailing they will be getting is an 
important survey and not "junk mail 1 ' to be tossed out 
unopened . 



WHAT SHOULD BE DONE AS THE COMPLETED QUESTIONNAIRES COME BACK? 

Each day, count how many questionnaires are returned. 
Compare the code numbers you put on the back of each question- 
naire against your master mailing list and check them off so 
these people won't receive a second mailing. 

You may want to mark each questionnaire with the date 
of receipt. This lets you look for differences in responses 
between early respondents and late respondents if that is of 
concern to you. 



WHAT. IS AN ACCEPTABLE RESPONSE RATE? 

One of the most common questions asked by novice re- 
searchers is, "What percent of my questionnaires do I need 
to get back?" There are both simple rnd complex answers to 
this question. The simple answer is that most pros like a 
70% to 80% response rate and consider 50% minimally adequate. 

The complex answer is that this is almost an irrelevant 
question for two reasons. The first reason is that, as any 
good researcher will tell you, quality of responses is more 
important than quantity . It is more important to have 
respondents who are generally representative of the group 
you are sampling from than to have a large return rate. 

Here's an example. Let's say you are conducting two 
surveys of your school's student body. Survey #1 yields 




46 



only a 35% response rate, but the respondents match the 
student body in terms of proportions of freshmen, sopho- 
mores, juniors, and seniors; proportions of men and women; 
and proportions of full-timers and part-timers. Survey #2 
yields a 65% response rate, but almost all the respondents 
are full-time juniors and seniors and are disproportionate- 
ly male. Most researchers would agree that Survey #1 has 
yielded "better quality" or more valid responses than Survey 
#2, despite its lower response rate. 

This is why it's important to collect demographic 
information on your sample, either from the data base you 
took your mailing list from or from a few questions at the 
end of the survey. 

The second reason that this question is almost irrele- 
vant is, well, suppose that despite your best efforts you 
only get a 30% or 40% response rate. After all the time and 
expense you've put in, you're not going to throw out those 
questionnaires, are you? You'll probably go ahead and 
analyze the data and simply note the low response rate as a 
caveat in your final report. 

Some of your subjects will undoubtedly have bad or 
missing addresses on file. Any subjects that you cannot 
locate should be deleted from your sample and not considered 
when you calculate your response rate. 



ARE FOLLOW-UP MAILINGS WORTH THE TIME AND EXPENSE? 

Definitely! A rule of thumb is that each follow-up 
will get you an additional 50% beyond your previous mailing. 
Thus, if you get a 40% response to your first mailing, you 
can expect an additional 20% from your second mailing and 
another 10% if you do a third — a total response rate of 70%. 
Follow-up mailings can therefore almost double your response 
rate . 

Another important reason for doing a follcw-up mailing 
is that late respondents sometimes differ from early ones. 
If this is true for your survey, conductin , only one mailing 
will bias your results. 



WHAT FORM SHOULD FOLLOW-UPS TAKE? 

Follow-ups can take many forms. You will need to select 
the appropriate format based on your budget and your need 
for a high response rate. Some possibilities: 

1 . Another complete questionnaire package : cover 
letter (amended to read, "We haven't heard from you yet!"), 
questionnaire, and return envelope. 
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2. A reminder postcard or letter, asking respondents 
to return the questionnaire package mailed earlier. This is 
cheaper than mailing a complete package but not as effective, 
since some people may have discarded or misplaced the 
original questionnaire. 

3. A complete questionnaire package sent by certified 
mail. This can be effective but expensive. Keep in mind 
that respondents may be inconvenienced and irritated at 
having to travel to the post office to sign for a "dumb 
survey" . 

4. A double tear-off postcard, asking the respondent 
to complete, tear off, and mail a postcard containing only 
the most crucial questions. This will generate a higher 
response rate than a long questionnaire, but preparing 
"cuctom" postcards like this can be expensive. 

5. A telephone call, either to remind subjects to 
answer the questionnaire or to ask at least some of the 
questions over the phone. The feasibility of this varies 
considerably depending on the nature of your telephone 
service (D^ you have a WATS line? Do students in your dorms 
have their own phones?) and the availability of staff to 
make calls. 

These formats can be used in combination. You could, 
for example, follow an initial mailing with a reminder 
postcard, then with another complete package, and finally 
with a certified mailing or a phone call. The basic rule: 
persistence pays off. 



WHEN SHOULD FOLLCW-UPS 3E MAILED? 

This depends on whom you are surveying and how long it 
will take for mail to reach them. If you are surveying 
students on campus, you can expect to receive questionnaires 
three days after you mail them. If you are surveying alumni 
throughout the country, you will need to allow a week for 
the questionnaires to get to them and another week for them 
to be returned. 

Here is an example of a follow-up timetable for a 
survey of currently enrolled students: 

Day 1 (Monday) : Mail initial questionnaire package. 

Days 2 and 3 (Tuesday and Wednesday) : Questionnaires 
are received by students. 

Days 4 and 5 (Thursday and Friday) : Keturns begin 
coming in. 



Day 8 (Monday) : Mail reminder postcards to everyone 
(thanking them if they've returned the questionnaire 
already) T 

Days 11 and 12 (Thursday and Friday): Prepare second 
mailing of complete package to non-respondents. 

Day 15 (Monday): Mail second questionnaire package. 

Day 2 4 (Wednesday): Begin telephoning non-r€>spondents . 



WHY NOT ADMINISTER THE SURVEY IN PERSON? 

When possible , administering a survey to your sample in 
person has a number of advantages: 

1. It is cheaper; you save on envelopes and postage. 

2. It is faster; you don't have to wait for the mails 
to come through. 

3. You will get a higher response rate. 

4. You will be able to answer unanticipated questions 
and problems. 

There are two problems with in-person administrations, 
however. The first problem is that you may want to survey a 
group that is not easily accessible in person: all under- 
graduates at your institution , alumni , or evening students , 
for example. Contacting your sample through classes, 
meetings, etc., will not give you a good random sample of 
your population. 

A second problem is that respondents may feel forced to 
complete something they would rather not do, and this may 
affect their responses. If your survey is on a sensitive 
topic, your respondents may think a colleague might see 
their responses as they are being completed and adjust their 
answers accordingly. A respondent who would like to write a 
lengthy response to a question may feel pressured if he or 
she sees everyone else has finished. 

In-person administrations are nonetheless suitable for 
a variety of situations, particularly evaluations and 
exercises collected as part of a class workshop or other 
group activity. 



HOW SHOULD YOU ADMINISTER AN IN-PERSON SURVEY? 

Here are some tips that will help an in-person adminis- 
tration yield good quality data: 
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1. Leave off the cover letter. Prepare instead a 
statement to read to the group explaining the nature and 
purpose of the survey and soliciting their help. 

2. If you are administering the survey to two or more 
groups, keep your directions identical from one group to the 
next. 

3. Repeat key parts of your oral statement at the top 
of the questionnaire. 

4. Establish an atmosphere where respondents do not 
feel trapped, rushed, or otherwise uncomfortable. Don't 
force anyone to complete the survey. 

5. If other people are helping administer the survey, 
train them carefully so that everyone administers the survey 
in the same way. Try to anticipate questions so everyone 
will give roughly the same answers. 

6. Provide a large envelope or box for respondents to 
put their completed questionnaires in and seal it immediately, 
so confidentiality is preserved. 



WHAT TASKS CAN BE DELEGATED TO SOMEONE ELSE? 

Once the questionnaire and cover letter have been final- 
ized, most of the tasks in this chapter can be delegated to 
a clerk or student. Clerks or students can do any of the 
following: 

1. Arrange for duplication of questionnaires, cover 
letters, and envelopes. 

2. Arrange for mailing labels for the outer envelopes. 

3. Prepare the first mailing, including numbering the 
master mailing list and questionnaires with code numbers. 

4. Check off respondents as returns come in. 

5. Prepare subsequent mailings. 

Depending on their skill and aptitude, clerks and stu- 
dents may also be trained to administer in-person surveys. 
They may also be able to edit and code responses and transfer 
data onto coding sheets (these are discussed in the next 
chapter) . 

Many surveys make ideal independent study projects or 
internships for students in business, mathematics, education, 
psychology, sociology, etc. Under your supervision, these 



students can do the entire survey (except perhaps for 
typing, duplicating, and keypunching) from beginning to end, 
including : 

1. (Most important) Developing a time line to make, 
sure the entire study can be completed in one semester or 
academic year. 

2. Designing and testing the questionnaire and cover 
letter . 

3 . Selecting the random sample and arranging for 
mailing labels. 

4. Preparing each mailing and checking off returned 
questionnaires . 

5. Editing and coding the returned questionnaires and 
writing the co<?ebook. This is covered in the next chapter. 

6. Writing a computer program to analyze the data. 
This is covered in Chapter VI (Analyzing the Survey Data) . 

7. Writing the final report. This is covered in the 
last chapter. 



WHAT CAN A WORD PROCESSOR OR MICROCOMPUTER DO? 

If you have access to one, a word processor or micro- 
computer can be an invaluable aid with the tasks described 
in this and earlier chapters: 

• Your questionnaire and cover letter drafts can be devel- 
oped and stored on a word processor or a microcomputer 
with word processing capabilities. 

• Your mailing list can also be stored and maintained on 
a word processor or microcomputer if it has the capa- 
bility to generate mailing labels as well as a roster 
of names and addresses. The microcomputer can assign 
ID numbers and can keep track of who has returned 
questionnaires . 

• As ncted earlier , word processing equipment can generate 
individually typed cover letters which may enhance your 
response rate. 

• There are now a few statistical software packages avail- 
able in PC versions (e.g., SAS IBM PC version, SPSS/PC, 
BMDPC) . If you have one of these and enough memory, 
you may be able to run your data analyses on your micro- 
computer. 



Chapter V. 



PROCESSING THE SURVEY RESULTS 



Once your questionnaires are being returned, you can 
begin to process the data to prepare it for statistical 
analysis. There are three basic steps to processing survey 
data: editing the responses, coding them, and getting the 
data "keyed" into a computer. You will then be ready to 
write a computer program to analyze the data, a task dis- 
cussed in the next chapter. 



WHAT SHOULD YOU LOOK FOR AS YOU EDIT RESPONSES? 

Some respondents will not have followed directions 
properly; others will have marked their answers unclearly or 
ambiguously. The first step in processing data is therefore 
to review every c ; ^stionnaire for inappropriate responses. 
Use a distinctly ±y colored pen to mark your corrections. 

Inconsistent responses. You may have a section that 
only Business majors were to answer, yet some other majors 
completed it. Cross out their responses to this section. 

Multiple responses . Even though you may clearly ask 
respondents to check the "ONE best response", some re- 
spondents may check two or more answers to a question. 
There are several ways to handle this: 

1. Code all responses. This is usually not preferable 
because multiple response data is difficult to handle. Also, 
in doing this you are giving unfair weighs to respondents 
who followed directions incorrectly. Had you given everyone 
this opportunity, other respondents may have checked more 
than one response too and changed your overall results. 

2. Delete all of the respondent's answers to the 
question. This is also often not preferable because you 
will lose data. From an empirical point of view, however, 
it is the most v lid way to handle this situation, because 
you have no way of knowing which one response the respondent 
would have selected had he or she followed directions cor- 
rectly . 

3. Choose one response at random and cross out the 
others. This is often preferable because you will salvage 
at least some information. Just make sure you choose the 
response randomly ; don't always code the first response 
checked. 

Responses outside given categories. Suppose you have 
some items with possible responses of "Excellent , 11 "Good," 
"Fair," and "Poor," and som one checks midway between 
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"Excellent" and "Good". Your choices here are either to 
delete the response altogether or to recode the response 
randomly to either "Excellent" or "Good". 

"Other" responses that really aren't. Some respondents 
will check an "Other — please specify" response when in fact 
one of the given responses suits them. Read all the "Other 
— please specify" comments and recode as necessary. 



WHAT IS CODING? 

Coding responses means changing every answer into a 
number, set of numbers, or some other character or symbol. 
A response of "yes" may be coded "1", for example, "no" may 
be coded "2", and "unsure" as "3". Coding is done because 
codes make it much easier for a computer to analyze data. 



HOW SHOULD RESPONSES BE CODED? 

Questions in which the respondent chooses the ONE best 
answer. Generally assign each possible answer a number and 
code accordingly. 

Example Code 

Question 7. What is your class level? 

Freshman 1 

Sophomore 2 

Junior 3 

Senior 4 

In this example, a respondent checking "Junior" would 
be coded "3". 

Questions in which the respondent checks all the 
answers that apply. For coding purposes, these questions 
must be treated as a series of yes/no questions. 

Example Code 

Question 8. Wnat influenced you to apply to Aloysius 
College? Check a* *_ that apply. 

High school guidance Question 8A. Code "1" if 

counselor checked . 

"College night" Question 8B. Code "1" if 

checked. 

Visit to campus Question 8C. Code "1" if 

checked . 

Admissions brochure Question 8D. Code "1" if 



checked. 
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Each response is coded as if it were a separate 
question. Checked responses are usually coded "1"; other 
responses are either coded "0" or left blank. 

Open-ended questions. Coding answers to these ques- 
tions (e.g., "What one thing do you like best about Northern 
University?") ? s a more subjective task than coding the 
questions discussed above. Here is a suggested guide: 

1. Quickly read through the responses and make a list 
of the categories they fall into. 

2. Read through them a second time, making sure each 
response can be assigned to one and only one category. 

3. Go through the responses a third time, this time 
coding them. 

4. Ask someone else to go through your coding and see 
if he or she agrees with it. 

To keep coding consistent, the entire set of responses 
to any one question should be coded by one person. The 
coding should then be reviewed by someone else to help pre- 
vent bias. 



WHAT GENERAL CODING PRINCIPLES SHOULD BE FOLLOWED? 
Here are some general guidelines: 

1. Use numbers rather than letters or symbols (for 
example, code "female" as "1" rather than "F") . Many 
computer programs cannot analyze non-numeric codes or can 
only do so on a limited basis. 

2. Leave blanks blank. Don't code them as zero or 
"999" . 

3. Avoid using zeros as codes. On some systems zeros 
and blanks are read the same way. 

4. Use identical codes for questions with the same set 
of responses. For example, if you code one question using 4 
for "strongly agree," 3 for "agree," 2 for "disagree," and 1 
for "strongly disagree," use this coding scheme for all such 
questions . 

5. Try to use numeric codes in a logical order. 
Coding "yes" as 1, "don't know" as 2, and "no" as 3 puts the 
answers into a continuum of sorts. 
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6. Don't code decimal points, dollar signs, coiunas, 
etc. This wastes coding time and computer space. The 
computer data analysis program can reinsert these symbols 
later. 

7. Don't collapse categories to simplify your data 
analysis. The computer can do this for you later. 

HOW ARE THE CODED DATA ENTERED INTO THE COMPUTER? 

To understand the whys and wherefores of getting coded 
data into a computer, let's look at common conventions for 
storing data in a computer: 

1. Generally all the data on one respondent should fit 
onto one line or computer card. Each line is usually 80 to 
120 spaces long. 

2. Each space on the line holds one digit, symbol, or 
character (like the spaces on a typewritten line) , Each 
space is called a "column 11 . 

3. Each answer is generally in the same location from 
one line to the next. If your code for Question 7 is in 
Column 27 for one respondent, for example, the codes for all 
respondents' answers to Question 7 should be in Column 27. 

These rules can be broken, especially with more modern 
computer systems, but to be safe it is best to follow them. 

Coded data are entered into the computer by data entry 
clerks who either keypunch the data onto cards to be fed 
into the computer or type the data directly into the computer 
at terminals. Data entry clerks work at very high speeds 
and, since they are probably unfamiliar with your work, must 
be able to read your codes explicitly. 

You can communicate ^our coded data to the data entry 
clerks in one of two ways: by transferring the data onto 
coding sheets for the data entry clerks to type from, or by 
putting data entry instructions directly on the question- 
naires and sending them directly to your Computer center for 
entry. We will discuss both. 



WHAT ARE CODING SHEETS AND HOW ARE THEY PREPARED? 

A ceding sheet (usually available in pads from your 
computer center) looks like a big piece of graph paper, 
usually 80 boxes across and 25 or so down. Data is written 
onto a coding sheet, one character per box, in exactly the 
format you want it entered into the computer. (Question 7 
codes in Column 27, Question 3 codes in Columns 28-29, etc.) 
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Before you start writing your coded responses onto 
coding slr^ts, take a few moments to design the sheet's 
layout: 

1. As mentioned earlier, try to fit each respondent's 
data on one line. 

2. Use the first few columns to code your respondents' 
ID numbers. Sequence your ID numbers starting with 101 
rather than 1. 

3. If you have more than one line of data per respondent, 
use the next column for the line number. 

4. Use the next columns to code data you have collected 
about the respondents other than their questionnaire responses 
(e-g., sex, major, geographic origin, grade point average). 

5. Use the remaining columns for questionnaire responses. 
Most questionnaire responses will need only one column each. 
Items with more than ten possible answers will need two columns. 
Items asking respondents to write in a figure (e.g., how much 
they spends on books each semester) may also need more than 

one column. Checklist or multiple response items need one 
column for each possible answer. 

6. Avoid blank columns between answers. Data entry 
clerks work much faster typing a string of digits than a 
series of digits and spaces. 

V7hen you transfer your daca onto coding sheets, use 
ink, not pencil — it's easier for the clerk tc read. (If you 
make a mistake, just draw a heavy line through that line of 
coding and start over on the next line.) Print neatly (a 
11 7 " can be confused with a "Y", for example) and use capital 
letters for all alphabetic codes. Drav; a slash through the 
letter 0 (0) to distinguish it from a zero and make sure the 
letter I is distinctly different from the number 1. Some 
people also drav: a line through the letter Z (2) to dis- 
tinguish it from a 2. 



HOW CAN DATA ENTRY INSTRUCTIONS BE PUT DIRECTLY ON A 
QUESTIONNAIRE? 

Many questionnaires can be printed with column and code 
numbers so that, after editing and perhaps coding a few 
open-ended items, the questionnaires can be sent directly to 
the computer center for data entry. Sometimes "op-scan" 
sheets can be used. The advantage of these approaches is 
that they save you and your staff considerable coding time. 
But, as discussed in Chapter III (Developing the Survey 
Instrument) , some respondents may think that their question- 
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naires will go directly to a computer and that no one really 
cares about their individual responses, particularly written- 
-in ones. Others may find the codes or op-scan format make 
the questionnaire confusing. For these reasons, use data 
entry codes or op-scan sheets in the limited circumstances 
described in Chapter III , 

There are other disadvantages to pre-coded question- 
naires and op-scan sheets. The chances for data entry error 
are greater. The temptation to skip editing is great, 
especially when using op-scan sheets which make editing 
difficult. And as you or your staff code data, you will 
undoubtedly catch problems that you missed when you edited 
the responses. Data entry clerks, most likely unfamiliar 
with your study, will not know how to deal with these pro- 
blems. They will therefore either misread and miscode the 
data or stop work until they can find you and ask you about 
the problem „ 

There is also something to be said for "getting your 
hands into the data," and nothing will do this like coding 
data yourself (or having a staff member familiar with the 
study do it) . Coding forces you to look at every response, 
giving you new insight into your data and helping you inter- 
pret your results. You may observe unanticipated inter- 
relationships among responses, for example, or notice 
firsthand that one item didn't quite work right and should 
be discarded. 

Pre-coded questionnaires are nonetheless useful in some 
circumstances. Codes may be printed on the questionnaire as 
follows : 

Multiple choice questions ; 

What degree are you seeking? 
(8) 

1 Not seeking a degree 

2 Associate degree 

3 Bachelor's degree 

4 Master 'f degree 

The data entry clerk would key in a 1, 2, 3, or 4 in 
Column 8, depending on ths response checked. 
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Checklists simply need the column numbers: 



In what business subjects would you be interested 

in taking courses? 

(34-39) 

Accounting 

Economics 

Finance 

Law and Business 

Management 

Marketing 



Here the data entry clerk would key in a 1 in Column 34 
if the respondent checked Accounting, a 1 in Column 38 if 
the respondent checked Management, etc. Unchecked spaces 
would be left blank. 

Grids like Likert scales and semantic differentials can 



also include coding instructions: 

How would you rate... Very good Good Fair Poor 

4 3 2 1 

Advisement? (17) 

The library? (18) 

The quality of the labs? (19) 

Faculty teaching ability? (20) 



Here the data entry clerk would key in a 1, 2, 3, or 4 
in each of Columns 17-20 depending on the response checked. 

If you decide to use a pre-coded questionnaire, first 
prepare a coding layout, just as if you were going to use 
coding sheets. Then mark the codes on your draft question- 
naire and ask a data processing expert to review it for 
feasibility and clarity. Be sure to provide space and 
instructions for entering ID numbers and other information 
you have collected about the respondents. 

A good data entry professional may be able to help you 
design a questionnaire that data can be entered from without 
pxe-printed coding instructions. Discuss this possibility 
with your computer center. 



WHAT IS A CODEBOOK? 

A codebock is a few pages on which the coding scheme 
and layout are documented . It is used to communicate this 



important information to your staff, your editors and coders 
the data entry clerks, and ycur programmers to minimize con- 
fusion and misunderstandings. It is also invaluable if you 
decide to repeat your study or do a follow-up on it. 

A codebook lists the name of every piece of information 
you are putting on the computer, which column (s) it is locat 
ed in, its format (numeric, alphabetical, etc.) and the 
meaning of ea^h code. Here is an 2xample: 



Data element 


cc 


Format 


Codes/Other information 


ID number 


1- 3 


999 




Sex 


4 


A 


M = male 
F = female 


Grade point avg 


5- 7 


9V99 




Question 1 


8 


9 


1 = Yes 

2 = Unsure 

3 = No 


Question 2 


9-10 


99 


Amount spent :>n food 



Some notes about this exa r ole: 



1. M cc" means Columns. Sex is in Column 4; Question 2 
is in Columns 9-10. 

2. The Format column uses "computerese " notation: 

" 9 11 s are used to show the data are numeric; a "V" is in- 
serted wherever a decimal point should go. This tells the 
computer programmer, for example, that a grade point average 
stored in the computer in Columns 5-7 as "326" should be 
read and printed as n 3.26" . 
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Chapter VI. 
ANALYZING THE SURVEY DATA 



The advent of computers and statistical computer "pack- 
ages" such as SPSS has brought a myriad of statistical analysis 
techniques to the fingertips of even the novice researcher. 
V7hile this capability gives the researcher a great deal of 
power, it also means he or she can be overwhelmed with the 
plethora of statistical analyses available. This chapter is 
design _d to help you make your choices. It has been written 
for those with at least a modest familiarity with statistical 
techniques. If you are a novice with statistics, we suggest 
you find a statistical consultant (perhaps a faculty member) 
to help you analyze and interpret your data. 

Before you choose yonr analyses, you need to know wh t 
kind of data you have: nominal, ordered, or interval (See 
Chapter I for explanation of those tc^rms.) You also need to 
know what information you would like from your data. In 
particular, you need to decide if you simply want to describe 
your group or if you want to try to explain why your group 
responded or behaved in a particular way. Describing is a 
simpler task for explaining, so we'll discuss that first:. 
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DO YOU SIMPLY WANT TO DESCRIBE YOUR GROUP? 



If you want to describe your group with a chart or graph 
of responses, use the chart below to choose the appropriate 
method. 



What kind of 

What do ycu want data do you Use this 

to do? have? method 



Examinp f hp rp^nnn<;p^ 

J— * dill ±. 1 ICOUUlloCO 

of one group to one 
question 


Interval or 
ordered 


Frequency 
distribution, bar 
gi. cipii f xxne grapn 


Nominal 


Frequency 
distribution , bar 
graph, line graph 


Compare the responses 
of two groups to one 
question 


Any kind 


Paired frequency 
distributions or 
graphs 


Compare the responses 
of one group to two 
questions 


Interval 


Scattergram 


Ordere" or 
nominal 


Cross tabulation 
table 



If you want to describe your group with statistics , use the 
chart below to choose the appropriate statistics. 



What do you want What kind of data Use this 

to describe? do you have? statistic 



The average 
response 


Interval 


Mean or 
median 


Ordered 


Median 


Nominal 


Mode 


The spread or 
variability 
of responses 


Interval 


Standard 
deviation 


Ordered 


Semi-inter- 
quartile 
range or 
range 


Nominal 


Proportion 
falling 
outside 
mode 


The degree of 
the relationship 
between two 
responses 


Interval 


Pearson 1 s 
product- 
moment 
correlation 
coefficient 


Ordered 


Spearman 
rank-order 
correlation 
coefficient 

Kendall tau 
coefficient 


Nominal 


Cramer 1 s 
index of 
contingency 


The degree of 
relationship 
among three or 
more responses 


Interval 


Multiple 
correlation 
Partial 
correlation 


Ordered 


Kendall 
partial 
rank 

correlation 



KOW WELL DO YOUR SAMPLE RESULTS CORRESPOND TO THE VIEWS OF THE 
ENTIRE STUDENT BODY? 



When researchers analyze survey data, we want to be able 
to assume that our sample results are similar to those we 
would get if we were able to survey every student. Suppose, 
for example, that we surveyed 400 students and found that 
58.2% of them are "satisfied" with the current registration 
process. We would like to be able to say that 58.2% of all 
students at our college are satisfied with the registration 
process . 

Assuming we surveyed a random sample of students, can we 
say this? No! It's very unlikely that exactly 58.2% of the 
entire student body is satisfied with the registration process. 
It's possible that the overall percentage is really 58.6%, 
57.9%, maybe even 63.1%. This discrepancy between our 58.2% 
and the true percentage is called sample error (first discussed 
in Chapter I.) It is not really an "error", just an phenomenon 
that exists because even a good random sample is unlikely to 
match precisely the results from the results from the entire 
group . 

When reporting your results, you wil?. sound much more pro- 
fessional and credible if you mention the possible sample error 
of your findings. For example, instead of simply saying "58.2% 
of students are satisiiied with the registration process, "say, 
"58.2! of students are satisfied with the registration process 
with an error margin of plus or minus 6% ." Statisticians know 
this means that you are 95% sure that between 52.2% and 64.2% 
of students are satisified with the registration process. 



HOW IS THE ERROR MARGIN CALCULATED? 

When are you reporting percents , this formula will give 
you a good estimate of the error margin: 



/ 1 x 100% where n = your sample size 
/ n 

In the example above, we surveyed 400 students so the 
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error margin was /400 x 100% = 1 ' 0025 x 100%=. 05 x 
100% = 5% 

While you can use this formula to find the error margin 
of any percentages you have, it will slightly overestimate the 
error margin of percents close to 0% or 100%. If you want a 
more accurate error margin in these cases, look in any statis- 
tics textbook for a discussion of "confidence intervals for 
proportions. 11 

fin 



Error margins can also be calculated for means, for dif- 
ferences between two means, and for differences between two 
proportions. To find out how, again look in any statistics 
textbook for "confidence intervals. 11 



DO YOU HAVE TO TRY TO EXPLAIN WHY YOUR GROUP RESPONDS OR 
BEHAVES IN A CERTAIN WAY? 

This is a more interesting kind of analysis than simply 
describing your results. Statistical analyses are available 
to help you answer the following kinds of questions: 

• Is my group difference from an established or theorized 
norm? 

• Is my sample truly representative of my population? 

• Are two or more subgroups different from each other? 

• Are two or more responses from my group different? 

• has ny group changed over a period of time? 

• Are there any interrelationships among my group's 
responses? 

The statistical analyses that answer these questions are 
hypotheses tests . Hypothesis tests follow five steps: 

1. An hypothesis is made about the results. To keep 
things simple, the hypothesis is usually that there is no 
difference, relationship, etc., even if you're actually hoping 
or expecting that there will be a difference. 

2. The hypothesis is initially assumed to be correct. 

3. A statistical analysis is done to find out the pro- 
bability of getting our sample results if the hypothesis is 
true. 

4. If there is a reasonable probability that we'd get 
our sample results, we conclude the hypothesis may be true and 
that there is no evidence of a difference, relationship, etc. 

5. If it's very unlikely (usually less tnan 5% chance 
or a 1% chance) that we'd get our sample results, either our 
sample results are wrong or the hypothesis is wrong. Since 
our sample data are from a well-planned, valid study (this is 
why it's important to have a valid study!), we conclude that 
the hypothesis is wrong and that there is a "significant" 
difference, rela. ionship, etc. 
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The rest of this chapter is a guide to choosing the 
statistical analyses most appropriate for your data and the 
questions you have about then. As we've mentioned before, if 
you are a statistical novice, you may want to ask a statistical 
consultant to help you choose your analyses and interpret them. 
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ESTABLISHED NORM OR A THEORIZED VALUE? OR DO YOU WANT TO SEE 
IF YOUR SAMPLE IS TRULY REPRESENTATIVE OF YOUR POPULATION? 

Your might want to see, for example, if students at your 
school differ from the national norms. 

If your data are interval , use d t-test for one mean* 

If your data are ordered , use a Kolmogorov-Smirnov one- 
variable test. 

If your data are nominal, use a chi-square test for good- 
ness of fit or a t-test for one proportion. 



DO YOU WANT TO SEE IF TWO OR MORE SUBGROUPS ARE DIFFERENT FROM 
EACH OTHER? 

You might want to see, for example, if there's a differ- 
ence between men and women or among freshmen, sophomores, 
juniors, and seniors. Use the chart below to choose the 
appropriate analysis. 
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What kind of data 
do you have? 



How many subgroups 
do you have? 



6'6 



Use this 
analysi s 



Interval 


Two 


t-test for two 

independent 

means 


Three or more 


one-way analysis 
of variance 


Ordered 


Two 


Mann-Whitney 
U test 


Three or more 


Kruskal-Wallis 
one-way 
analysis of 
variance 


Nominal 


Two 


t-test for two 
proportions 


Two or more 


chi-square test 
of association 



DO YOU WANT TO SEE IF TWO OR MORE RESPONSES FROM YOUR GROUP 
ARE DIFFERENT? OR DO YOU WANT TO SEE IF YOUR GROUP HAS 
CHANGED OVER A PERIOD OF TIME? 

You might want to compare for example, your student's 
latings of the math department and the computer science 
department to see if they differ. Or if you have been 
following a group of alumni over a period of time, you might 
want to compare their perceptions five years ago. Use the 
chart below choose the appropriate test. 



How many responses 
What kind of data do you want to Use this 

do you have? compare? analysis 



Interval 


X WW 


f-f pef f of 

matched pairs 


Three or more 


one-way analysis 
of variance 


Ordered 


Two 


sign test 

Wilcoxon matched 
pairs si ed 
ranks test. 


Two 


Friedman two-way 
analysis of 
variance 


Nominal 


Two 


McNemar test for 
significance of 
changes 


Three or more 


Cochran Q test 



DO YOU WANT TO LOOK FOR INTERRELATIONSHIPS AMONG YOUR GROUP'S 
RESPONSES? 

The analysis you choose will depend on the kind of data 
you have and the kind of relationship you wish to examine. 
Sometimes researchers are interested in the int3rrelationship 
among two or more responses. More often, however, they are 
particularly interested in the relationship of one or mora 
responses to one particular response or factor. A researcher 
might want to study, for example, the relationship of several 
questionnaire responses to : 
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• Grade point average 

• Whether or not a student drops out 

• Whether or not a student flunks out 

• Responses to one question asking for overall 
satisfaction with Northern College 

DO YOU WANT TO STUDY THE RELATIONSHIP OF SEVERAL RESPONSES 
TO ONE PARTICULAR RESPONSE OR FACTOR? 

You can use multi /ariate analysis of variance, re- 
gression analysis, or discriminant analysis to do this. 

If you want to study the relation of two or more nominal 
responses to one interval response, use multivariate analy- 
sis of variance. This analysis would b3 used, for example, 
if you wanted to study the relation of sex, age group, and 
major to grade point average. 

If you want to study the relation of one or more interval 
responses to another interval response, use regression analy- 
sis. This analysis could be used, for example, if you wanted 
to study the relation of several irterval responses to re- 
sponses to one question asking for overall satisfaccion with 
college . 

If you want to study the relation of two or more interval 
responses to one nominal (usually bivariate) response, use 
discriminant analysis. ^his analysis would be used, for 
example, if you used several interval responses to distin- 
guish between students who drop out and students who return. 

If all the data you're studying are interval, you have 
a real choice between analysis of variance and regression 
anal^ois. Consider tY ,e points as you make your decision: 

Analysis of variance is actually a broad term covering 
many different kinds of analyses. You can look for differ- 
ences among several responses made by one group, differences 
in responses among several subgroups, or simultaneously 
examine differences wiLhin and among subgroups. You can 
also use Scheff£ contrasts to see how the responses affect 
the response you're most interested in. For more informati n 
on these kinds of analyses, consult a textbook ca experi- 
mental design . 

Regression analysis restricts you to one group but gives 
you a lot more information. In particular, it lets you predict 
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the response of most interest to you from the other responses 
you're studying. You mighc predict grade point average, for 
example, from responses to several interval questions. Re- 
gression analysis will also let you measure the accuracy of 
your predictions and qi\r you information on the relative 
importance of several responses to the response of most 
interest to you. 



DO YOU SIMPLY WANT TO EXAMINE THE RELATIONSHIP AMONG TWO OR 
MORE RESPONSES? 

You can use correlation tests, chi-square tests, or 
factor analysis to do ihis. 

If you want to see if there is a relationship between 
two interval responses, use Fisher's z test for the Pearson 
correlation coefficient. 

If you want to see if there is a relationship between 
two ordered responses, use a test of significance for 
Spearman's correlation coefficient. 

If you want to see if there is a relationship between 
two nominal responses, use a chi-square test of association. 

If you want to study the interrelation among a large 
set of interval responses, use factor analysis. It "reduces" 
the responses into factors or components. Researchers use 
factor analysis to (1) explore relations among responses, 
(2) test hypotheses or theories about what the interrelation- 
ships should be, and (3) reduce a large number of responses 
to a manageable few for further study. 



HOW SHOULD CENSUS DATA BE ANALYZED? 

Virtually all the statistical analyses presented in 
this chapter are inferential statistics, designed to be used 
to make an inference from a sample to an entire population. 
What if you have conducted a census, collecting data on 
everyone in your qroup rather than just a sample? There are 
three schools of thought on how to handle this: 

1. Since the data are from a census, they are completely 
accurate. Any difference, however small, is a real difference 
and therefore significant. No inferential statistics — hypothe- 
sis tests or error margins — are called for. 

2. Although the data are from a census and all dif- 
ferences are therefore statisticall y significant, some 
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differences may not be large enough for practical signifi- 
cance. Hypothesis tests and error margins help eliminate 
the very small differences with no practical significance. 

3. Although the data are from a census at one point in 
time, they are being used to make inferences about slightly 
different future populations. A census of the Fall, 1988 f 
student body, for example, might be used to make decisions 
affecting the Spring, 1989, student body, which would be 
somewhat different. Under such circumstances, the Fall, 
1988, student body would be considered a sample of all 
students, present and fucure . Inferential statistics 
including hypothesis tests and error margins are therefore 
appropriate . 



WHAT ELSE SHOULD YOU CONSIDER AS YOU CHOOSE YOUR ANALYSES? 

Perhaps the most important thing you should keep in 
mind is that even statistical experts often disagree on the 
most appropriate analysis for a given set of data. Don't 
let this chapter dictate what to do. If you have doubts 
about the appropriate analysis for a particular situation, 
consult some statistics textbooks or someone knowledgeable 
•^bout statistics. 

Another thing to keep in mind is that the analyses 
listed here can be used with "higher level" data. Analyses 
suggested for nominal data, for example, can be used with 
ordered or interval data. Analyses suggested for ordered 
data can be used with interval data as well. 

This often gives you a real choice among analyses. In 
most cases, the preferable analysis will be the parametric 
one designed for interval data. Parametric analyses usually 
have the following advantages : 

1. They are the most well known . Yc-<r reader is more 
likely to be familiar with analysis of variance than the 
Cochran Q test, for example. Using a familiar test will 
make your results easier for your reader to understand. 

2. They usually provide tuore in format ion . Most of 
them, for example, are designed to provide information on 
why individuals vary from the mean. This information can be 
very helpful in understanding your results. 

3. They are more powerfu l. This means that when you 
decide that there is no significant difference, relationship, 
etc., you have a better chance of being right than if you've 
used an analysis designed for nominal or ordered data. 
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4. They are nore robust or flexible. While they are 
usually based on assumptions about the data, the assumptions 
can often be violated without affecting the validity of your 
results. Many of these analyses can be used, for example, 
with ordered data even though they are designed strictly for 
interval data. Analysis of variance technically assumes 
that each set of responses you are comparing comes from a 
separate group, but it is robust enough that you can use it 
to compare three or more responses from one group. 

Don't forget, by the way, th*t bivariate data can be 
treated as interval data. You can often convert one nominal 
response into a ^et of bivariate (yes/no) responses in order 
to use an analysis designed for interval data. 

A final note: Many of these analyses have limitations 
and restrictions to their use beyond what could be gone into 
here. Before you use any of these tests, read up on them in 
a statistics textbook to make sure the ones you choose are 
appropriate to use with your data. 



WHAT ELSE SHOULD GO INTO THE DATA ANALYSIS COMPUTER PROGRAM? 

The data analysis program should have several additional 
elements : 

1. A listing of the raw data. This will help you find 
and correct coding and punching errors. 

2 . Edit checks . Always ask for frequency distributions 
of every response. This will give you a quick picture of 
your data and help you identify bad data or codes. You may 
also want to ask for two-way tables ("cross-tabs"), showing 
the frequency distributions of two items together, to identi- 
fy inconsistent answers. For example, you might want to 
compare sex and participation in men's and women's sports. 

3. Reliability and validity checks. Making reliability 
and validity checks of your items was discussed in Chapter 
III (Developing the Survey Instrument) . Include a check on 
the "representativeness" of your sample respondents. Plan 

to compare the characteristics of your respondents (e.g., 
proportions of men, freshmen, business majors) with known 
characteristics of the group you took your sample from. 

4 . Data transformations . You will want to make a few 
minor changes to your data before you analyze them: 

• Identify missing data. Blank or missing data should be 
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clearly identified and then deleted from data analysis. 
An important exception: blanks in checklists are valid 
responses and should not be identified as missing. 

• Reverse inversely coded items. If you have a Likert or 
semantic differential scale, probably some items are 
positive and others are negative. If you coded the 
response columns 1, 2, 3, and 4, have the computer 
reverse the codes of the negative items (changing 1 to 

4, 2 to 3, 3 to 2, and 4 to 1) . This will make it 
easier to compare responses. 

• Create new variables. You may want to "collapse" some 
categories that few people fell into, add up several 
numbers to a total, and so on. 

5. Data labeling and documentation. Provide plenty of 
labels, headings, and so on so your printout will be easy to 
read. Put in enough notes that your data transformations 
will be easily understood. 



WHAT "PACKAGES" ARE AVAILABLE TO ANALYZE THE DATA? 

A large number of statistical "packages" are available 
to analyze data on a large "mainframe" computer or mini- 
computer, including SPSS, SAS, BMDP, and Minitab. Inquire 
which are available at your computer center. Most of these 
are easily self-taught, especially if you have ever done any 
programming , and there is almost certainly someone on your 
campus familiar with them and able to help you. 

As discussed in Chapter V (Processing the Survey Re- 
sults) , there are also statistical software packages avail- 
able for microccmputers . Since they are generally inter- 
active, you do not actually "write" the program beforo you 
run it, but you should still plan what you will do before 
you sit down at the terminal. If you have access to both a 
microcomputer and a mainframe, keep in mind that there are 
two limitations to using a microcomputer for analysis of 
survey data: 

1. The microcomputer may not have enough memory to 
store all your data or to ru r * more sophisticated statistical 
analyses such as factor analysis. 

2. The data will probably need to be typed in by you 
or your staff rather than d data entry clerk, and it is a 
tedious, time consuming job. 

Microcomputer software is therefore best for analyzing 
the results of relatively small surveys, such as workshop 
evaluations or surveys of students in just one department. 
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HOW SHOULD THE DATA ~*E STORED? 

Your data will need to be stored in such a way thct you 
preserve the confidentiality of your respondents 1 answers 
and don't betray their trust. There are actually t iree sets 
of data to be stored securely: the master mailing list, the 
questionnaires themselves, and the coded data. 

The master mailing list, the one that relates res- 
pondents' names to their ID numbers (and thus links names 
with responses), should be stored very securely. Don't 
leave it on a computer file if there is any chance of an 
unauthorized person accessing it. 

The questionnaires should be saved because the ■ .ten- 
in responses will be useful in interpreting data ana. .es, 
writing the final report, and answering questions from 
people who read the final report. If the questionnaires 
have identifying information on them (e.g., social security 
numbers) , they should also be stored securely. 

Computerized data may be stored on computer tape, disk, 
or cards, depending on the prjference of the computer center. 
Once the data have been thoroughly checked and edited, you 
may want to ask to have the ID numbers "stripped" (deleted) 
to guarantee the confidentiality of the data even further. 
If you are using a microcomputer to analyze your data, put 
the data on a floppy disk and store it in a locked cabinet 
when you're rot using it. 
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Chapter VII. 



REPORTING THE SUR/EY RESULTS 



The final aspect of conducting a survey is to com- 
municate the findings clearly and accurately so that they 
can be used for decision making. Your report will need to 
be planned as carefully as any other part of the survey 
research process, considering who your audience will be and 
what its needs are. 



WHO WILL READ AND USE YOUR FINDINGS? 

Before you begin any writing, you should know whom you 
are addressing. Ask yourself: 

1. Will your readers already be familiar with what 
you have been doing, or will they need a complete descrip- 
tion of what was done? 

2. Do they have time to study an extensive report or 
will they want only a short summary? 

3. Will they want only your findings and recommenda- 
tions or will they want to know how you arrived at your 
conclusions? 

4. Are they knowledgeable about empirical research 
methodology or will you need to explain what ycu did in 
layperson 1 s language? 

5. Are they likely to criticize the study? Will you 
need to anticipate their criticisms and incorporate re- 
sponses into your report? 

6. Are they likely to be questioned about the study 
by others? Will they need sufficient details to respond 'co 
others 1 concerns? 



IN WHAT FOR' I DO YOUR READERS NEED THE FINDINGS? 

As you plan and write your report, you will have to 
wrestle with two seemingly contradictory needs of your 
readers: that your report be the essence of brevity yet 
include all the details they consider important. Your 
orimary challenge will be to reconcile these two disparate 
.*eeds . 

Here are some of the forms your report may take, 
depending on which is more important to your readers — their 
reed for brevity or their need for information — c.nd how 
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knowledgeable and interested they are in empirical 
research: 

A few tables summarizing your findings. If your 
readers are already familiar with your study, it may be 
sufficient to make up a few tables highlighting your 
principal findings and pass them around. But some people 
have difficulty understanding tables, and your charts may 
not be all that clear. In such cases, a couple of brief 
explanatory paragraphs accompanying your tables may make a 
much better report. 

The executive summary , This is a one or possibly two 
page summary of your study. It's a distant cousin of the 
abstract used in scholarly research. The major difference 
is that, while an abstract summarizes the entire study, the 
executive summery emphasizes your findings, especially those 
that will be of most interest to decision makers. 

An executive summary should be the first page of any 
lengthy report sent to busy people. It can also be sent 
out alone, with an offer to provide the complete report on 
request (you'll be surprised how few people take you up on 
your offer) . 

The executive summary may be widely read and quoted. 
It's therefore important to select carefully the points to 
be communicated in the summary and to do so in a form that 
is attractive and easy to understand. 

A complete report. Most survey research projects re- 
quire a complete report that documents what you did, why 
you did it that way, and all your findings and conclusions. 
Some of your readers will want a complete report, if only 
to have on file or to skim for parts of interest to them. 
Even if che complete report is sent to no one, it should be 
prepared and kept in your files to answer any quescions and 
to serve as a resource should anyone decide to conduct fur- 
ther research on the topic. If your study was fun'ed by 
anyone, the sponsor should get a complete report. 

A short report. Sometimes a shorter version of the 
full report is needed to provide more detail than the 
executive summary without burdening readers with a full 
report. If your complete report runs more than ten pages, 
you'll piobably need a short version of it to get your 
message across to decision makers. 

Supplemental reports. These are an excellent way to 
get some of the nuances and details of the study to only 
those people who would be really interested. For example, 
suppose you are doing a survey of student satisfaction with 
your college. You might send only to the student life 



staff details of the findings on satisfaction with dorm 
life. You might send only to the computer center staff or 
computer science faculty details of the findings on satis- 
faction with computer access, 

An oral presentation. This can be a very effective 
way to share your study with people who want only high- 
lights and don't like dealing with tables of statistics or 
dry text. You may have difficulty, however, in convincing 
the "powers that be n to give you time to make your presen- 
tation. Try distributing an executive summary first and 
then asking to make a presentation. 

If you make an oral presentation, remember that even 
though you found your project fascinating in every detail, 
your audience probably won't. By keeping your remarks 
short and informal, concentrating on your findings, and 
allowing for questions, you will be able to hone in quickly 
on the areas of major concern to your audience. (If they 
want to know more about how you did the survey, they'll 
ask.) A few handouts or overhead transparencies will be 
appreciated, provided they are kept simple, uncluttered, 
and readable. 

More than one report. You may need to plan different 
reports for different audiences. You may prepare a com- 
plete report with an executive summary for chief decision 
makers, a shorter report to the faculty discussing only 
those findings of direct interest to them, a very brief 
summary for the public relations office to use in press 
releases, and an oral presentation to a committee or task 
force . 



WHAT ARE THE COMPONENTS OF A SURVEY RESEARCH REPORT? 

A complete report should include each of the following 
elements : 

A meaningful title. Often the title alone must con- 
vince your audience to read your report. Your title should 
therefore explain what your report is about and help con- 
vince them to read it. "Student Survey Results" says 
nothing; "Factors Related to Student Attrition" says much 
more. Questions ("Why Do Students Drop Out?") can pique 
readers' curiosity. (Make sure they can quickly skim the 
report and find the answer!) Some of the best titles con- 
dense the principal findings of the study: "The Impact of 
Faculty/Student Interactions in Reducing Student Attrition". 

An executive summary gives a quick overview of the 
purpose of the study and hov; it was done and highlights 



principal findings and implications. Be careful to reflect 

the results accurately in this summary. The temptation to 

oversimplify or overreach for meaning is great and, as noted 

earlier, the executive summary is likely to be widely cir- 
culated and quoted. 

Tne purpose of the study. To orient your reader, you 
should explain why the study was done and what it was 
designed to find out. This may be only a sentence or two. 

Depending on the nature of your study and your audience, 
you may want to include some background information to help 
orient your reader further. If you are doing a survey re- 
lated to student attrition, for example, you may want to 
give a brief summary of the literature on factors related 
to student attrition. If you are doing a survey related to 
some other campus problem, you may want to give a brief 
history of circumstances leading up to the study. 

Hov; the study was done. Write at least a few sentences 
explaining how your data were collected. The following 
points are crucial for the reader to decide on the validity 
and usefulness of your results and should be included in 
any report, short or long: 

1. The group you took your sample from (e.g., all 
full-time freshmen as of Fall, 1987) . 

2. How the pjrsons contacted w^re selected. 

3. The number of people contacted, the number 
responding, and the response rate. 

4. Any evidence that the people who responded are an 
unbiassed sample of the group you wanted to survey. 

5. Whether the questionnaire was distributed by mail 
or in person. 

6. The date the survey was conducted. 

7. Any caveats that readers should keep in mind as 
they interpret the findings. These might include notes 
about a low response rate, a response group that is not 
quite representative of the overall group you wanted to 
survey, or anything that has happened since the survey was 
conducted that might affect results were it repeated today. 

Results , This is the part of your report that most 
readers will concentrate on. Since it is also the most 
complicated, it requires careful thought and planning so 
that it may be clearly understood by all your readers, 



regardless of their knowledge of research nieluods ar*i 
statistical analyses . 

Your readers will find your results easier to follow 
if they are broken into clearly labeled sections: 

An overall descriptive summary of your results. This 
is the part of your report that most readers wilx concen- 
trate on. It therefore requires careful writing to keep it 
clear. 

Since fl a picture is worth a thousand words" , the best 
way to communicate your basic findings is to include a copy 
of the actual questionnaire with the results filled in. If 
you don't include the actual questionnaire, the questions 
that produced the results should be quoted exactly, since 
small differences in wording can produce large differences 
in responses. 

For many items on the questionnaire, you'll want to 
fill in the percent who checked response A, the percent who 
checked response B, and so on. Do not give the number of 
people who checked A and the number who checked B; these 
figures are meaningless. Convert your counts into per- 
cents; most people are more used to dealing with them than 
decimals . 

For some questions ycu may want to fill in a mean or 
median response. It is particularly helpful to report mean 
or median responses from rating scales. This lets sharp 
readers quickly compare responses to find out which item 
had, say, the highest rating. Sc;ae readers may similarly 
appreciate standard deviations, since these tell them which 
items had the most consistent and inconsistent ratings. 

Regardless of whether you report means, medians, or 
percents, round your figures to the nearest whole number. 

readers will care that 10.3% said X and 10.4% said Y; 
many will appreciate the increased readability of rounded 
statistics. To the "mathophobes" in your audience, overly 
precise statistics merely increase the number of figure 
they must examine. Unrounded numbers also encourage un- 
sophisticated readers to focus on trivial differences. 

You might want to include some "typical" responses to 
open-ended questions in this section. 

V7hile this first section should be statistically 
simple, it should include the margin of error discussed in 
the previous chapter. This figure is essential for your 
readers to judge which differences are truly meaningful. 
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More detailed results. If your results are at all 
extensive, the next sections of your report should give 
your findings for subgroups, findings of interest to only a 
few readers, and special analyses such as correlations 
among responses and results of statistical tests. 

These sections should be clearly labeled so that 
readers can skip them if they like. Headings should 
describe the point of the analysis ("Factors Affecting 
Grade Point Average") rather than the analysis itself 
("Results of Multiple Regression Analysis") . 

Unless your entire audience is expert in statistics, 
describe any sophisticated statistical analyses (e.g., 
analysis of variance) in layperson's terms. 

Avoid the temptation to report every single statisti- 
cal analysis you conducted. If it doesn't mak< a mean- 
ingful contribution to your conclusions, leave it out. 

These are not the sections for subjective dicussions 
or interpretations. Although people may be interested in 
your opinions about the results, they have the right to be 
able to look at the pure data, ursullied by anyone's be- 
liefs, and draw their own conclusions. 

Tables and graphs. You will probably do a much better 
job of communicating your findings to your readers if you 
present some of the data in tables or graphs. A well- 
constructed table will permit the reader to quickly and 
easily find and compare figures of interest to him or her. 

A picture is worth a thousand words, however, only if 
it is a good one! Most tables and graphs are so poorly 
designed that they hinder rather than help understanding of 
the data. Here are some tips: 

1. Give each table and graph a self-explanatory 
title, "Responses to Question 21" won't do it* 

2. Label each part of the table or graph clearly . 
Label each table column and each graph c.xis. Avoid ab- 
breviations and avoid writing labels vertically. 

3. Since some readers will read the table or graph 
without reading the text, make it self-explanatory. Give 
definitions, assumptions, and notes in footnotes at the 
bottom of the table. 

4. Avoid puttinc* too much information into one table. 
If you find you must insert vertical lines into your tabic 
to make it clear, ycu have too much data. Break your 



information into two or more tables, or simply delete some 
of it from your report. 

5. Don't confine yourself to presenting your results 
in the order of the original questionnaire. Present checK- 
list or multiple choice results ranked with the most fre- 
quently chosen answer at the top, where the reader can 
quickly find it. 

6. Don't assume a computer-generated table or graph 
is readable. Some computer programs generate distorted 
tables or graphs; most provide poor labels if any; some 
include bits of "computerese" the c can't be deleted. 

For an example of a good format for a clear table, see 
the American Psychological Association's Publication Manual. 

Conclusions and recommendations . In this section, 
draw your conclusions from your findings and your statisti- 
ca analyses. Your conclusions should clearly relate your 
da ~a to the questions addressed by your study and described 
in the introduction to the report. 

This section may also include the following: 

1. Speculations on possible reasons why your findings 
turned out the way they did. Do uftis only if you don't 
mind being wrong! Discuss your ideas with colleagues before 
you include them to make sure they make sense. 

2. Recommendations for actions that might be taken 
based on your findings. Institutional researchers have 
strong disagreements about whether it is their place to 
make recommendations. Recommendations do help drive home 
the major conclusions from your study, but they can also 
create problems for you. By implying that something can be 
improved, recommendations imply that: something is presently 
not as good as it should be, and this is very likely to 
offend whoever is responsible for that area. Unfettered 
recommendations can result in challenges about the quality 
of your research and a serious undermining of your credi- 
bility . 

If you don't want your recommendations to get you in 
trouble : 

• Praise as well as criticize. Include a positive 
conclusion for every problem you uncover. 

• Keep your recommendations uncontroversial ("More 
encouragement for student independent study projects 
is needed . " ) . 
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• Confxrm an already accepte. problem ("More parking 
space is urgently needed."). 

• Phra?e your recommendations gently. Rather than say 
"The college should provide more activities on week- 
ends," say ; "Further research may be needed into the 
appropriateness of the college's weekend activities." 

• Don't phrase your recommendations in such a way that 
anyone reading them might feel threatened or defensive. 
Avoid pinning blame on any one department. 

• If your study unearths a problem in a particular area, 
talk to the person in change of that area before yovr 
report is released. At the very least you will have 
forewarned che supervisor. More likely you will gain 
some insights that help explain your findings and 
possibly change your recommendation . The supervisor 
may tell you he or she is awau of the problem and is 
delighted to see support from you for resources to 
help address it. 

• If some of your conclusions or recommendations are 
very sensitive (if, for example, you identify evidence 
of sex discrimination in a particular department) , 
omit the most sensitive details from your written 
report and instead discuss them in person with those 
who should be aware of the problem. Remember that 
your study only provides allegations of a problem, not 
proof cf it. 

3. Suggestions for further research . Any good study 
raises more questions than it answers. Note if you are 
planning on studying these issues further. You may have 
some ideas for someone else whc would be interested in 
conducting further research. I call this section "Looking 
Ahead" and it's a nice way to end the report. 



HOW ELSE CAN YOU KEEP YOUR REPORT INTERESTING AND READABLE? 

This is not a journal article. While your readers are 
probably intelligent and well-educated, chances are they 
are not familiar with research methodology, statistics, or 
the subject of your study. Even if they are. they probably 
don't have time to study a scholarly missive. Here are 
some suggestions for ways to improve the readability of 
your report. 

Use plenty of headings , so busy executives :an scan 
your report and read only what interests them. 
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Keep your sentences cind paragraphs short. If one 
sentence runs six or eighc lines or one paragraph fills 
most of a page, your report is too difficult to read. 

Avoid research jargon such as "variable , 11 "subject," 
or "population." Keep your writing simple, clear, and in 
layperson's terms. It ^s possible, for example, to explain 
the results of regression £ ;alysis without ever using the 
term "regression" . Avoid statistical symbols; write 
"standard deviation" instead of "s", tor example. 

Spell out apbreviations at least the first time they 
are used, even if they are widely understood on your 
campus. Also spell out any numbers* under 13. 

Minimize citations . Since this is net a scholarly 
paper, you need not include fill citations, although you 
should of course explain in some less formal way the 
sources of any ideas vhat are not your own. You probably 
do not need to include a reference list or bibliography; 
anyone who would like a complete citation can contact you. 

As a fina] ch3ck for readability, have a friend 
outside your field read your report. Even if he or she 
knows no statistics, your friend should be able to follow 
the purpose of your study, your basic findings, and your 
conclusions and why you drew them, 

These suggestions are not meant to imply that your 
report should have a casual tone. Your challenge is to 
keep it clear and simple while maintaining its dignity. 
Avoid slang, contractions, and undignified expressions 
("really interesting") . Avoid writing in first person 
("I", "we") or second person ("you"), although writing in 
the first person is increasingly acceptable if it stream- 
lines and clarifies your report. 
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Postscript 
REACTING TO OTHER SURVEYS 



Although the focus of this monograph has been on 
conducting your own survey, x hope that you will keep the 
points made here in mind when you are asked to complete 
other questionnaires or when you read reports on other 
surveys. A let of surveys, both good and bad, are being 
done these days. We have a responsibility to support good 
surveys. Return the favors your respondents did for you by 
answering any good questionnaires you receive. 

We have a similar responsibility to contravene bad 
surveys. Bad surveys hurt us. Past frustrations with them 
is one of the reasons that potential respondents refuse to 
help us. Bad surveys also warte the time we spend attempt- 
ing to complete them, since tie information gathered will be 
largely useless. 

By refusing to cooperate with bad surveys, we not only 
stop wasting our time but may also contribute to a very low 
overall response rate. Perhaps that will make the researcher 
and his or her readers think twice about using the results 
or about conducting similar surveys in the future. 

If you would like to discourage poor survey reseaich, 
toss any questionnaire with any of the following character- 
istics right in the trash. 

1. No indication of who is sponsoring the survey. 

2. No explanation of how the results will be u'ocd. 

3. No guarantee of confidentiality. 

4. No clear definitions of ambiguous terms. (I once 
received a survey asking for "faculty workload" data How 
iuany different definitions must that have!) The person 
conducting the survey will end up with apples-and-oranges 
data that's useless. 

5. No postage-paid envelope in which to return the 
survey . 

6. A multi-page survey that will take a lot of time to 
complete (searching through files, calling other offices), 
unless the uso it will be put to clearly justifies the time. 
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7. Any obviously biased or loaded questions. This 
includes requests for factual data that will obviously 
support only one side of an issue. 

We also have an obligation to critically review any 
reports on surveys before reacting to them or taking actions 
based on them. The Canadian Daily Newspaper Publishers 
Association has published a "journalist's checklist 1 ' to 
evaluate polls. Much of it is applicable to other survey 
reports. . ny report you read should include the following: 

1. Who sponsored the survey? If someone with a vested 
interest in the results was the sponsor, the survey may be 
biased by biased questions, a biased sample, or a selective 
disclosure of the results. 

2. When was the survey conducted? Has anything happen- 
d in the interim that might change results if the survey 

were repeated today? 

3. How were the persons polled selected? What evidence 
is there that they are an unbiased sample of the desired 
population? 

4. Row many persons responded? 

5. What was the response rate? 

6. //hat were tne questions? Small differences in 
wording can produce large differences in responses. The 
questions that produced the results should be quoted exactly. 

7. What is the error margin? 

3. Does the executive summary of the report accurately 
reflect the results? The temptation to oversimplify or over- 
reach the meaning is great. 

I once worked for a short time as a radio announcer. 
When I was learning the ins and outs of radio, someone re- 
marked to me, "Now that you know what v ^ do, you'll never 
listen to radio the same way again." He was right; I will 
never again be able to listen as idly (or ignorantly) as I 
once did. I cannot help but be sensitive to what the an- 
nouncer says, the "rotation" of the songs played, even the 
content and length of the commercials. 

The story applies to survey research. Now that you 
knt>w how to conduct a good survey, I hope you will never 
react in the same way to anyone else's questionnaire or 
report. Even if you never write another questionnaire 
yourself, our time will not have been wasted. 
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Complete? 





Start 


Tire 


Task 


date 


need go 


Plan Survey. 




2 weeks 


Review what others have done. 




2 weeks 


Write questionnaire and cover letter. 




1 week 


Pilot test questionnaire. 




1 week 


Design data analysis. 




1 day 


Arrange for data entry. 




— 


Write data analysis computer software. 




1 week 


Obtain mailing labels. 




1 week 


Select sample. 




2 days 


Collect additional data as needed. 




2 weeks 


Type and duplicate questionnaires, cover 




2 weeks 


Jetters, and return envelopes. 






Stuff envelopes for first mailing and mail. 




2-3 days 


Wait for returns. 




2-3 weeks 


Prepare follow-up mailing and mail. 




2-3 days 


Wait for returns. 




2-3 days 


Edit responses and begin data entry. 




1-3 weeks 


Analyze data. 




2-3 days 


Write report. 




1 week 


Have report typed and distributed. 




1 week 



Deadline 



Note. These items need not be completed sequentially. For example, the data analysis can be 
designed and the mailing labels ordered \;hile the questionnaires are being printed. 
Additional data can be collected while you are waiting for the questionnaires to be returned. 
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Appendix 2 . 
EXAMPLES OF QUESTIONNAIRES 

This appendix includes the follov/ing examples of question- 
naires, all written by the author. References to specific 
institutions have been deleted. 

"Survey of Student Goals and Satisfaction" 

"Summer Orientation Program Student Evaluation" 

"Survey of Alumni Satisf act ton , Activities, and Goals" 

"Office of Institutional Studies User Survey 1 
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OFFICE OF THE PRESIDENT 

A UNIVERSITY 

SURVEY OF STUDENT GOALS AND SATISFACTION 



Unles • otherwise requested, che»; the ONE best response of each item. If 
you want to explain your answers further, use the space at the end of the 
questionnaire or attach another ; x iece of paper. 

1. What is your goal here at A University? 

To finish a degree here. 

To prepare to transfer to another college before finishing 

a degree here. 

(What other college? ) 

At this time I'm not sure about my goal here. 

Other (Please specify ) 



How would you describe A to a friend back horn*-? 

Check the part of each line that best shows your feelings. 



Beautiful campus 



Unattractive campus 



Too expensive Reasonably priced 



Good academic reputation 
Good location 
Too big 
Good financial aid 
Too close to home 



Weak academic reputation 
Poor location 
Too small 

Little financial aid 
Toe far from home 



3. How would you describe A 1 s courses to a friend back home? 

Check the part of rich line that best shows your feelings about the 
courses here in general . 

Interesting Boring 

Usually closed East to get into 

Conveniently scheduled Inconveniently scheduled 

Wide variety Small selection 

Easy Hard 

Intellectually challenging Mostly busy work 




4. How would you describe A 1 s faculty to a friend back home? 

Check the part of each line that best shows your feelings about the 
faculty here in general . 

Cold and remote Friendly 

Always available Hard to find 

Hard to talk to __ Easy to talk to 

Willing to help Unwilling to help 

Hard to understand Easy to understand 

Enthusiastic "Burned out" 

PLEASE CONTINUE ON OTHER SIDE. . . 

BO 



5. How would you descril your academic advisor to a friend back home? 
Check the* part of each line that best shows your feelings* 



/"".ways available 
Hard to talk to 
Willing to help 
Cold and remote 
Gives good advice 



Hard to find 
Easy to talk to 
Unwilling to help 
Friendly 

Gives poor advice 



6* Kow would you describe your social life at A 



to a friend 



back home? Check the part of each line that best shows your feelings. 



Nothing to do 
Hard to make friends 
Good athletics programs 
Poor facilities 



Lots to do 
Easy to make friends 
Weak athletics programs 
Good facilities 



7. Below are listed some opinions that students sometimes express about 
college* Check all the statements Xhat reflect your feelings* 

University is a prestigious name in my community* 



The program I'd like to major in here is not as good as I'd 
like. 

The program I ! d like to major in is too hard here. 

I want to major in a program not offered here (What program? 

) 

I'm pretty good friends with a faculty member here. 

I may have to leave A because my grades aren't high 

enough. 

I'm running into more financial problems than I expected when I 
came here. 

I'd like vo take a brea*. from college for a year or so. 
I have a hard time finding computer terminals available. 
L.^ne of these statements applies to me. 



8. What do you plan to major in? 



9. What one thing do you like best about A_ 
specific ! 



Universe fy? Be 



10. What one thing about A 
Be specific! 



University most needs improvement? 



Thank you very much for your help. Please return this questionnaire in the 
addressed, postage-paid envelope provided. 
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Today's date 

Your guide's name 

Your intended major 



A_ University 

SUMXER ORIENTATION PROGRAM 
STUDENT EVALUATION 



We would like to know how successful the orientation program has been in famil- 
iarizing you with life at A University. How worthwhile was each part 

of the orientation program? For each question, please check the space that be t 
reflects your opinion. If you'd like to make any additional comments, please 
use the space at the end of this form. Thank you very much for your help. 

How worthwhile was each of the following programs cr presentations in 
acquainting you with A and preparing you to attend here? 



Very Somewhat Not Did Not 

Program/presentat * ?r Worthwhile Worthwhile Worthwhile Attend 

Firrt Day 

1. Welcomes from Orientation 
Committee and University 

administration 

2. Placement and proficiency 

testing 

3. Campus tour 

4. "Somebody Else's Problem" __ 

5. Intramurals 

6. Resident Life 

7. Commuter Life 

8. Parent's Club (evening) 

9. Student Guide and Video 

(evening) 

10. Awareness workshop (evening) __ 

11. Evening entertainment 



Second Day 

12. Undecided/undeclared students 

13 . ROTC 

14. Finarcial Aid 

15. Library 

16. Administrative Services 

17. Student Affairs 

18. Pre-registration academic 
advisement 

19. Registration 



UP 
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For each of the following statements, please mark the space that best matches 
your feelings. 



Strongly Strongly 
Agree Agree Disagree Disagree 

20. The check-in process went 

smoothly. 

21. My guide was knowledgeable. ~ZZ 

22. My guide was friendly. ZZZ 

23. My guide was helpful. __ ZZZ 

24. The food was good. 

25. The residence hall accommodations 

were clean and comfortable. 

26. The academic advisor I saw 
before registration was 
helpful. 

27. The registration process 

went smoothly. 

28. Overall, the presentations — _. 
were easy to understand. 

29. Overall, the presentations — 
were interesting. 

30. Overall, the people 1 met were ~~~ 
helnful. 

31. Overall, these two days have — — — - - 
been worthwhile. 



32. What was the one best part of the entire program? 



33. What one part of the program most needs improvement? 



How does it need improvement? Be specific! 



Additional comments: 



Thank you very much for your help! Please place this form in the box at the 
front of the room before you leave. 



ERJC 



o J 



Office of Institutional Research 
B College 
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Survey of Alumni Satisfaction, Activities, & Goals 



Unless otherwise requested, check the ONE best response to each item. If you want to 
explain your answers further, use the space at the end of the questionnaire or attach 
another piece of paper. 

1. If you could start over again, would you still go to college? 
Yes, and I'd go to B . 

Yes, but I'd go somehwere else (Where? ) 

No. 

2. How much did your education at B contribute to your growth in your 

ability to: 

Very A Not at 
much Somewhat little all 



Define problems 
Use a typical adacemic library 
Understand others 1 id^as through reading 
Understand other's ideas through 

listening 
Appreciate how various fields of 

study are interrelated 
Distinguish fact from opinion 
Understand and appreciate different 

viewpoints on a given topic 
Understand and apply scientific 

principles and methods 
Use widely-accepted research techniques 
Evaluate alternative solutions to 

a problem 
Organize your ideas 

Effectively explain your ideas orally 
Effectively explain your ideas in 

writing 
Perfoi a basic computations 
Appreciate the diversity of American 

culture 
Appreciate other cultures 



3. Since you graduated from B , have you participated in any of these 

activities ON YOUR OWN, NOT AS PARI OF YOUR WORK OR SCHOOL ASSIGNMENTS? 
Check as many as apply. 

Browsed in a bookstore 

Had a library card 

Worked on a crossword or similar kind of puzzle 

Discussed a national or world news event with friends 

Attended an art museum or exhibit 

Read an article on scientific advances 

Attended a play or opera 

Set up a family budget 

Kead a book dealing with a social or political issue 

Completed income tax forms on your own 

Read a "classic 11 literary work 

Attended a scientific exhibit 
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4. About how many hours per week do you read for leisure? 
hours per week 

5. If you are presently employed, which of these do you need to do your job well? 
Check as many as apply, 

I am not presently employed. 

Familiarity with methods of problem definition and solution 

Skill in critical thinking 

Ability to synthesize or merge knowledge from several sources 

Proficiency in your major field of study 

Acquaintance with other fields of study 

Basic skills in: 

Reading 

Writing 

Speaking 

Computation 

6. Have you ever applied for admissions to one or more graduate degree programs? 
Yes, and I was accepted into at least one program. 

Yes, but I was never accepted. 

Yes, but 1 haven't heard from them yet. 

No. 



7. What are the highest degrees beyond a Bachelor's that you've completed and 
that you plan to complete someday? 

Have Plan to 

Completed to Complete 

None beyond a Bachelor's 

Master's 

Doctorate (Ph.D., Ed.D. , etc.) 

Professional degree (i.e., doctorate 

in medicine, law, or theology) 

Other (Please specify ) 

I'm not sure. 



8. Have you taken or do you plan to take any non-degree courses since graduating 
from B ? Check all that apply. 

Haven't taken 

Have taken bat plan to 

Adult education (non-credit) 
Continuing Education (earning 

C.E.U. Credits) 

College credit courses required to 

keep/advance my position at work 

College credit courses for leisure 



9. Based on youi experience with B , what one thing about it most needs 

improvement? Be specific! 



Thank you again for your help. Please return this questionnaire in the enclosed 
postage-paid envelope. 
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OFFICE OF INSTITUTIONAL STUDIES 



USER SURVEY 



The Office of Institutional Studies is evaluating its services. We would 
appreciate your help by filling out this form. Check the appropriate responses 
to each item. Feel free to elaborate or comment on any answer. Please return 
this form to this office by October 27, 19 



1. Please checl; the frequency with which you use the services of 
Institutional Studies. 

Very rarely About once About once 
or never a term a month 



Once a week 
or more 



I receive "regularized 11 

reports 
I request oral reports or 

information 
I request written ad-hoc 

reports or studies 
I request consultant services 

(in DBMS, statistics, 

research methodology, etc.) 



Other 



2. Please check the quality of the services you used in terms of: 



Excellent 



Good 



Adequate 



Needs 
Improve- 
ment 



Accuracy 

Comprehensiveness 

Clarity /readability 

Timeliness 

Usefulness/value 

Overall quality of services 



3. Please c-:eck the quality of the staff in terms of: 



Knowledge 

Communication ability 

Helpfulness 

Courtesy 

Overall quality of staff 



Excellent 



Good 



Adequate 



Needs 
Improve- 
ment 



4. Additional Comments: 
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Appendix 3. 
EXAMPLE OF A COVER LETTER 

This appendix is an example of a cover letter, written by 
the author. References to a specific institution have been 
deleted . 



[Letterhead Paper] 



Date 



Dear Student: 

University is committed to providing you 

with quality educational opportunities. One way to see if 
we are meeting our goals is to ask % u how satisfied you are 

with f s various programs and features. Your 

input is very important in deciding what actions to take 

better to serve students who attend and thooe 

who plan tc come. 

Your thoughts are particularly important in these 
critical t^mes of budget cutbacks and uncertainties. 
Sharing youj opinions and perceptions with us is perhaps the 
most valuable contribution you can make toward helping us 
address these problems and take action to solve them. 

You are one of a small sample of students that was 
randomly selected to respond to a few questions on your 

goals and your satisfaction with . The small 

sample makes your response critical to the value of this 
study . 

Would you please take a few minutes to complete the 
enclosed questionnaire? An addressed envelope (postage-paid 
if you live off campus) is included for you to return it. 
The number cn the questionnaire will be used to contact and 
remind those not returning the questionnaire. Your name, of 
course, v/ili never be connected with yovr individual 
answers . 

If you have any questions about this study or if you would 
like a copy of the final report on it, please contact [name, 
title, address , telephone. ] 

Thank you for your help. 

Sincerely, 



[Name ] 
President 



Enc . 
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